[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 477641221 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003126] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007012] pid_max: default: 32768 minimum: 301 [ 0.008129] LSM: Security Framework initializing [ 0.009057] Yama: becoming mindful. [ 0.010035] SELinux: Initializing. [ 0.011070] *** VALIDATE selinux *** [ 0.019188] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023534] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026109] *** VALIDATE tmpfs *** [ 0.027416] *** VALIDATE proc *** [ 0.029220] *** VALIDATE cgroup *** [ 0.030009] *** VALIDATE cgroup2 *** [ 0.032281] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035026] Spectre V2 : User space: Vulnerable [ 0.036008] Speculative Store Bypass: Vulnerable [ 0.038727] debug: unmapping init [mem 0xffffffff9f259000-0xffffffff9f260fff] [ 0.040210] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041826] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042032] ... version: 2 [ 0.043027] ... bit width: 48 [ 0.044019] ... generic registers: 4 [ 0.045019] ... value mask: 0000ffffffffffff [ 0.046021] ... max period: 00007fffffffffff [ 0.047024] ... fixed-purpose events: 3 [ 0.048021] ... event mask: 000000070000000f [ 0.050285] rcu: Hierarchical SRCU implementation. [ 0.052738] smp: Bringing up secondary CPUs ... [ 0.053639] x86: Booting SMP configuration: [ 0.054037] .... node #0, CPUs: #1 #2 #3 [ 0.058308] smp: Brought up 1 node, 4 CPUs [ 0.060018] smpboot: Max logical packages: 1 [ 0.061025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.230019] node 0 deferred pages initialised in 167ms [ 0.233193] devtmpfs: initialized [ 0.234264] x86/mm: Memory block size: 128MB [ 0.237102] gcov: version magic: 0x41383552 [ 0.240213] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.244082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.246303] pinctrl core: initialized pinctrl subsystem [ 0.249296] [ 0.249703] ************************************************************* [ 0.251012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.254013] ** ** [ 0.256011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.258012] ** ** [ 0.260009] ** This means that this kernel is built to expose internal ** [ 0.261011] ** IOMMU data structures, which may compromise security on ** [ 0.263012] ** your system. ** [ 0.266014] ** ** [ 0.268013] ** If you see this message and you are not debugging the ** [ 0.270015] ** kernel, report this immediately to your vendor! ** [ 0.272024] ** ** [ 0.275013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.277011] ************************************************************* [ 0.278710] NET: Registered protocol family 16 [ 0.280625] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.283094] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.286170] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.291087] cpuidle: using governor menu [ 0.294022] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.297755] PCI: Using configuration type 1 for base access [ 0.300195] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.312077] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.314119] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.319069] cryptd: max_cpu_qlen set to 1000 [ 0.323329] ACPI: Added _OSI(Module Device) [ 0.325049] ACPI: Added _OSI(Processor Device) [ 0.327023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.328020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.333488] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.341378] ACPI: Interpreter enabled [ 0.343090] ACPI: PM: (supports S0 S3 S4 S5) [ 0.344074] ACPI: Using IOAPIC for interrupt routing [ 0.347158] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.350387] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.360946] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.363048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.365018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.367091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.372548] acpiphp: Slot [2] registered [ 0.374149] acpiphp: Slot [5] registered [ 0.375125] acpiphp: Slot [6] registered [ 0.376114] acpiphp: Slot [7] registered [ 0.377126] acpiphp: Slot [8] registered [ 0.378137] acpiphp: Slot [9] registered [ 0.379194] acpiphp: Slot [10] registered [ 0.381151] acpiphp: Slot [3] registered [ 0.382107] acpiphp: Slot [4] registered [ 0.383098] acpiphp: Slot [11] registered [ 0.384172] acpiphp: Slot [12] registered [ 0.385110] acpiphp: Slot [13] registered [ 0.387094] acpiphp: Slot [14] registered [ 0.388109] acpiphp: Slot [15] registered [ 0.390105] acpiphp: Slot [16] registered [ 0.391105] acpiphp: Slot [17] registered [ 0.393106] acpiphp: Slot [18] registered [ 0.395106] acpiphp: Slot [19] registered [ 0.396142] acpiphp: Slot [20] registered [ 0.398106] acpiphp: Slot [21] registered [ 0.400107] acpiphp: Slot [22] registered [ 0.401104] acpiphp: Slot [23] registered [ 0.403109] acpiphp: Slot [24] registered [ 0.405111] acpiphp: Slot [25] registered [ 0.406230] acpiphp: Slot [26] registered [ 0.408111] acpiphp: Slot [27] registered [ 0.410107] acpiphp: Slot [28] registered [ 0.411104] acpiphp: Slot [29] registered [ 0.413184] acpiphp: Slot [30] registered [ 0.415104] acpiphp: Slot [31] registered [ 0.416067] PCI host bridge to bus 0000:00 [ 0.418028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.421035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.423024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.425022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.428023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.430028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.432193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.435089] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.438172] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.447018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.451521] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.454019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.455015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.457017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.459588] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.463864] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.466045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.469836] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.475014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.491016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.498027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.504477] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.511017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.516015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.531023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.549854] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.560017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.572018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.591021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.601989] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.610017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.620014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.642014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.655110] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.663017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.670017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.691903] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.700185] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.708014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.714015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.731017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.746540] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.756015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.765021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.798015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.817545] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.819335] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.822703] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.824248] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.826140] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.830041] iommu: Default domain type: Passthrough [ 0.831355] SCSI subsystem initialized [ 0.833135] ACPI: bus type USB registered [ 0.834125] usbcore: registered new interface driver usbfs [ 0.837072] usbcore: registered new interface driver hub [ 0.838070] usbcore: registered new device driver usb [ 0.840197] pps_core: LinuxPPS API ver. 1 registered [ 0.842010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.845114] PTP clock support registered [ 0.847175] EDAC MC: Ver: 3.0.0 [ 0.849327] PCI: Using ACPI for IRQ routing [ 0.851469] NetLabel: Initializing [ 0.853012] NetLabel: domain hash size = 128 [ 0.854010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.856092] NetLabel: unlabeled traffic allowed by default [ 0.858241] vgaarb: loaded [ 0.860138] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.861011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.865555] clocksource: Switched to clocksource kvm-clock [ 0.982626] VFS: Disk quotas dquot_6.6.0 [ 0.984296] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.987262] *** VALIDATE ramfs *** [ 0.988686] *** VALIDATE hugetlbfs *** [ 0.990758] pnp: PnP ACPI init [ 0.994260] pnp: PnP ACPI: found 6 devices [ 1.013666] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.016885] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.018670] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.020879] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.023183] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.025750] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.028437] NET: Registered protocol family 2 [ 1.031233] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.034961] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.038223] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.043698] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.046714] TCP: Hash tables configured (established 65536 bind 65536) [ 1.049138] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.051572] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.053875] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.056354] NET: Registered protocol family 1 [ 1.058726] RPC: Registered named UNIX socket transport module. [ 1.061380] RPC: Registered udp transport module. [ 1.063426] RPC: Registered tcp transport module. [ 1.065421] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.068545] NET: Registered protocol family 44 [ 1.070895] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.073991] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.076779] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.079677] PCI: CLS 0 bytes, default 64 [ 1.082342] Unpacking initramfs... [ 2.470396] debug: unmapping init [mem 0xffff8be17cc54000-0xffff8be17ffbffff] [ 2.477424] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.480340] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.485223] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.989929] Initialise system trusted keyrings [ 2.991981] Key type blacklist registered [ 2.995332] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.004871] zbud: loaded [ 3.009817] *** VALIDATE nfs *** [ 3.011407] *** VALIDATE nfs4 *** [ 3.013216] pstore: using deflate compression [ 3.017628] Platform Keyring initialized [ 3.136210] NET: Registered protocol family 38 [ 3.139060] Key type asymmetric registered [ 3.140670] Asymmetric key parser 'x509' registered [ 3.142710] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.147410] io scheduler mq-deadline registered [ 3.149601] io scheduler kyber registered [ 3.152191] io scheduler bfq registered [ 3.154290] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.157701] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.161074] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.164225] ACPI: Power Button [PWRF] [ 3.169893] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.176908] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.192465] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.200302] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.217850] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.246016] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.276169] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.281777] Non-volatile memory driver v1.3 [ 3.284189] Linux agpgart interface v0.103 [ 3.315116] virtio_blk virtio1: [vda] 149496 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.318319] vda: detected capacity change from 0 to 76541952 [ 3.334169] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.337283] vdb: detected capacity change from 0 to 1073741824 [ 3.350748] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.353980] vdc: detected capacity change from 0 to 2621440000 [ 3.368637] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.372037] vdd: detected capacity change from 0 to 2621440000 [ 3.385679] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.389967] vde: detected capacity change from 0 to 4294967296 [ 3.404343] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.407876] vdf: detected capacity change from 0 to 4294967296 [ 3.416290] libphy: Fixed MDIO Bus: probed [ 3.425868] usbcore: registered new interface driver usbserial_generic [ 3.429229] usbserial: USB Serial support registered for generic [ 3.431707] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.436710] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.438718] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.441377] mousedev: PS/2 mouse device common for all mice [ 3.445949] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.451093] rtc_cmos 00:05: RTC can wake from S4 [ 3.458507] rtc_cmos 00:05: registered as rtc0 [ 3.460665] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.464314] intel_pstate: CPU model not supported [ 3.465139] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.473094] hid: raw HID events driver (C) Jiri Kosina [ 3.473200] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.475198] usbcore: registered new interface driver usbhid [ 3.482608] usbhid: USB HID core driver [ 3.484499] drop_monitor: Initializing network drop monitor service [ 3.487396] Initializing XFRM netlink socket [ 3.489615] NET: Registered protocol family 10 [ 3.492478] Segment Routing with IPv6 [ 3.494263] NET: Registered protocol family 17 [ 3.496557] mpls_gso: MPLS GSO support [ 3.502144] RAS: Correctable Errors collector initialized. [ 3.504573] AVX version of gcm_enc/dec engaged. [ 3.506632] AES CTR mode by8 optimization enabled [ 3.593768] sched_clock: Marking stable (3593744762, 0)->(4475571907, -881827145) [ 3.597304] registered taskstats version 1 [ 3.599219] Loading compiled-in X.509 certificates [ 3.600946] zswap: loaded using pool lzo/zbud [ 3.624097] Key type big_key registered [ 3.636654] Key type encrypted registered [ 3.638893] ima: No TPM chip found, activating TPM-bypass! [ 3.641436] ima: Allocated hash algorithm: sha1 [ 3.643480] ima: No architecture policies found [ 3.645525] evm: Initialising EVM extended attributes: [ 3.647341] evm: security.selinux [ 3.648650] evm: security.ima [ 3.649884] evm: security.capability [ 3.651471] evm: HMAC attrs: 0x1 [ 3.654286] rtc_cmos 00:05: setting system clock to 2026-08-22 06:17:31 UTC (1787379451) [ 3.660178] debug: unmapping init [mem 0xffffffffa0203000-0xffffffffa03fffff] [ 3.663350] debug: unmapping init [mem 0xffffffff9ef82000-0xffffffff9f258fff] [ 3.672192] Write protecting the kernel read-only data: 28672k [ 3.676837] debug: unmapping init [mem 0xffffffff9d603000-0xffffffff9d7fffff] [ 3.679843] debug: unmapping init [mem 0xffffffff9df14000-0xffffffff9dffffff] [ 3.714972] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.725689] systemd[1]: Detected virtualization kvm. [ 3.727939] systemd[1]: Detected architecture x86-64. [ 3.729869] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.758824] systemd[1]: No hostname configured. [ 3.760993] systemd[1]: Set hostname to . [ 3.763667] random: systemd: uninitialized urandom read (16 bytes read) [ 3.766865] systemd[1]: Initializing machine ID from random generator. [ 3.931764] random: systemd: uninitialized urandom read (16 bytes read) [ 3.935329] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.940769] random: systemd: uninitialized urandom read (16 bytes read) [ 3.943624] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.952267] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.595038] device-mapper: uevent: version 1.0.3 [ 4.597993] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.332829] virtio_net virtio0 ens2: renamed from eth0 [ 5.353768] scsi host0: ata_piix [ 5.380715] random: fast init done [ 5.393306] scsi host1: ata_piix [ 5.396561] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.399279] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.740519] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.156403] random: crng init done [ 10.158883] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.607821] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.800226] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.120230] SELinux: Disabled at runtime. [ 12.184191] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.194420] systemd[1]: Detected virtualization kvm. [ 12.196557] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.747468] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.751425] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.755702] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.761246] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.765442] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.773116] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.784657] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.958392] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.650805] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.640036] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.845498] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.350964] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.467663] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (7s / no limit)[ 21.054185] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 21.713398] NFS: Registering the id_resolver key type [ 21.726595] Key type id_resolver registered [ 21.730943] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg356-server login: [ 80.612693] libcfs: loading out-of-tree module taints kernel. [ 80.671967] Key type ._llcrypt registered [ 80.675453] Key type .llcrypt registered [ 80.816373] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 99.471931] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 101.307399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 101.326079] alg: No test for adler32 (adler32-zlib) [ 102.879384] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 103.842831] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 105.600177] Key type lgssc registered [ 107.242987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 120.065977] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 135.322319] hrtimer: interrupt took 7033741 ns [ 162.911295] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 178.378615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 178.408244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.707531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 179.755588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 179.923964] Lustre: lustre-MDT0000: new disk, initializing [ 180.061576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.093522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 185.217701] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.423524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.548774] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 198.596602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 198.600517] Lustre: Skipped 1 previous similar message [ 198.745473] Lustre: lustre-MDT0001: new disk, initializing [ 198.841450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 198.869421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 198.886088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 203.084285] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.738077] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 216.885592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.123652] Lustre: lustre-OST0000: new disk, initializing [ 217.127354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 217.137567] Lustre: 8452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.206848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.455105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.473578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.618190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 223.152465] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.717665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.825474] Lustre: lustre-OST0001: new disk, initializing [ 237.831146] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.838070] Lustre: 9526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.904512] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.269371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 243.283364] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 243.339847] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 243.516880] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.385550] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 262.177635] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 268.698694] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing check_logdir /tmp/testlogs/ [ 273.158345] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing yml_node [ 277.370723] Lustre: DEBUG MARKER: Client: 2.17.54.116 [ 279.719811] Lustre: DEBUG MARKER: MDS: 2.17.54.116 [ 281.995787] Lustre: DEBUG MARKER: OSS: 2.17.54.116 [ 283.524462] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 02:22:09 EDT 2026 [ 297.989516] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 299.588090] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 314.848961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.850510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.863391] Lustre: Skipped 1 previous similar message [ 314.877591] Lustre: Skipped 3 previous similar messages [ 319.969638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 321.078190] Lustre: server umount lustre-MDT0000 complete [ 324.357111] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379772 with bad export cookie 8113074901192319453 [ 324.359181] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 324.369997] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 325.089025] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.089841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.101737] Lustre: Skipped 3 previous similar messages [ 325.111820] Lustre: Skipped 3 previous similar messages [ 329.126156] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.843051] Lustre: server umount lustre-MDT0001 complete [ 334.373476] Lustre: server umount lustre-OST0000 complete [ 337.607346] Lustre: server umount lustre-OST0001 complete [ 343.629433] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 351.451042] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 382.036599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 382.235841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 382.263611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 382.346754] Lustre: lustre-MDT0000: new disk, initializing [ 382.436764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.449973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 386.788224] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 396.078667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.194553] Lustre: 15076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 396.266361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 396.269624] Lustre: Skipped 1 previous similar message [ 396.321106] Lustre: lustre-MDT0001: new disk, initializing [ 396.438187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 396.477809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 396.485630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 400.076587] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 408.520911] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 413.854876] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 421.495036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 421.902943] Lustre: lustre-OST0000: new disk, initializing [ 421.908995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 421.921638] Lustre: 17043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 422.063628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.958408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 422.981416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 423.155914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 428.620934] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 438.329921] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 443.268627] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 443.429498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 447.859732] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 448.133892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 454.119888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 454.134323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.140735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 454.150455] Lustre: Skipped 1 previous similar message [ 454.161389] Lustre: Skipped 2 previous similar messages [ 464.352935] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 464.357192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 464.380260] Lustre: Skipped 3 previous similar messages [ 464.536160] Lustre: server umount lustre-OST0000 complete [ 467.938273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 467.950637] LustreError: Skipped 1 previous similar message [ 467.955710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.649805] Lustre: server umount lustre-MDT0000 complete [ 474.593472] LustreError: 15087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.620187] LustreError: 15087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 477.159557] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379925 with bad export cookie 8113074901192320986 [ 477.173317] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 477.175860] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 477.732273] Lustre: server umount lustre-MDT0001 complete [ 481.978777] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 02:25:27 (1787379927) [ 484.737470] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 486.720762] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 02:25:31 (1787379931) [ 494.360252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.085737] LustreError: 19454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.229328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 499.994146] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 500.707478] LustreError: 19455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.807741] LustreError: 19454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.851573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.922545] LustreError: 19455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.425171] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 523.926270] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 528.828330] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 535.462764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 542.868945] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 552.866501] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 560.615132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 560.617719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.626415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 560.626423] Lustre: Skipped 3 previous similar messages [ 560.658826] Lustre: Skipped 2 previous similar messages [ 566.860566] Lustre: server umount lustre-OST0000 complete [ 571.875359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 571.884062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.459865] Lustre: server umount lustre-MDT0000 complete [ 580.943743] LustreError: 19436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380028 with bad export cookie 8113074901192321861 [ 580.957614] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 581.318704] Lustre: server umount lustre-MDT0001 complete [ 593.569149] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 596.680562] Key type lgssc unregistered [ 597.014478] LNet: 22987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.019919] LNetError: 22987:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 597.045917] LNet: Removed LNI 192.168.203.156@tcp [ 597.979283] Key type .llcrypt unregistered [ 597.983531] Key type ._llcrypt unregistered [ 619.573690] Key type ._llcrypt registered [ 619.580283] Key type .llcrypt registered [ 619.699943] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 620.551349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 620.813581] alg: No test for adler32 (adler32-zlib) [ 621.902541] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 622.217300] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 623.960172] Key type lgssc registered [ 625.005821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 632.940213] loop: module loaded [ 633.050270] loop0: detected capacity change from 0 to 204800000 [ 633.349226] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.365768] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.612130] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.436185] loop0: detected capacity change from 0 to 204800000 [ 638.700379] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.713575] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.723204] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 643.181024] loop0: detected capacity change from 0 to 204800000 [ 648.809513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 648.822023] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.057546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 650.165769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 650.302376] Lustre: lustre-MDT0000: new disk, initializing [ 650.410719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.442114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 654.467187] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 662.998185] loop1: detected capacity change from 0 to 204800000 [ 667.748642] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 668.080187] Lustre: lustre-OST07c6: new disk, initializing [ 668.084788] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 668.090408] Lustre: Skipped 1 previous similar message [ 668.096255] Lustre: 26173:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 668.190226] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 674.741979] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 676.998385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 677.010915] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 677.046939] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 679.433783] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 679.445842] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 679.455685] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 679.465180] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 679.471580] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 679.480278] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 696.324668] Lustre: 25159:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 696.332256] Lustre: 25159:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 696.338321] Lustre: 25159:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 696.346512] Lustre: 25159:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 696.353599] Lustre: 25159:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 696.360875] Lustre: 25159:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 701.189457] Lustre: 25160:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 701.199666] Lustre: 25160:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 701.208151] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 701.217848] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.226480] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 701.238321] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.244794] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 701.250657] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.255502] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 701.260896] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.266684] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 701.275574] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.496990] Lustre: 25160:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 704.513481] Lustre: 25160:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 704.523761] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 704.531315] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.542142] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 704.549777] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.556845] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 704.564856] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.572442] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 704.583840] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.596866] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 704.606470] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.906297] Lustre: 25160:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 708.913974] Lustre: 25160:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 708.917206] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 708.921091] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.926211] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 708.932845] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.938642] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 708.948455] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.954438] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 708.959329] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.969038] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 708.978977] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.918524] Lustre: 26181:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 719.934974] Lustre: 26181:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 719.940087] Lustre: 26181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 719.949901] Lustre: 26181:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 719.956472] Lustre: 26181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 719.961028] Lustre: 26181:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 719.965717] Lustre: 26181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 719.970110] Lustre: 26181:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 719.974083] Lustre: 26181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 719.979575] Lustre: 26181:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 719.987957] Lustre: 26181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 719.992727] Lustre: 26181:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 768.996791] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.054504] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 773.956987] Lustre: server umount lustre-OST07c6 complete [ 785.681773] Lustre: server umount lustre-MDT0000 complete [ 803.579503] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 806.333790] Key type lgssc unregistered [ 806.690532] LNet: 28732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.700249] LNetError: 28732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.717704] LNet: Removed LNI 192.168.203.156@tcp [ 807.518238] Key type .llcrypt unregistered [ 807.520715] Key type ._llcrypt unregistered [ 826.012276] Key type ._llcrypt registered [ 826.018155] Key type .llcrypt registered [ 826.120536] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 827.367408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.458412] alg: No test for adler32 (adler32-zlib) [ 828.859798] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 829.119398] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 830.912328] Key type lgssc registered [ 832.639937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 841.446203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 841.492893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 842.969795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.592182] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 852.455060] LustreError: 29997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 852.493140] LustreError: 29997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 857.415755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.582026] LustreError: 29998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 858.036588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 863.442530] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 870.913902] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 877.040329] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 884.938318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 885.378540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 887.471048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 891.849298] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 899.510995] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 918.505328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 918.511391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 918.528159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 918.530898] LustreError: Skipped 1 previous similar message [ 918.543890] Lustre: Skipped 1 previous similar message [ 919.500236] Lustre: server umount lustre-OST0000 complete [ 923.622040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.635292] Lustre: Skipped 1 previous similar message [ 923.640598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 923.647516] Lustre: Skipped 1 previous similar message [ 928.740970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 928.751817] Lustre: Skipped 1 previous similar message [ 929.795968] Lustre: server umount lustre-MDT0000 complete [ 933.862134] LustreError: 29998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.868618] LustreError: 29978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380382 with bad export cookie 6993058499883956626 [ 934.872640] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 934.892101] LustreError: 29978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 935.239522] Lustre: server umount lustre-MDT0001 complete [ 949.853463] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 953.592942] Key type lgssc unregistered [ 953.994924] LNet: 33737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.009311] LNetError: 33737:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.032071] LNet: Removed LNI 192.168.203.156@tcp [ 955.025990] Key type .llcrypt unregistered [ 955.030862] Key type ._llcrypt unregistered [ 976.892661] Key type ._llcrypt registered [ 976.897768] Key type .llcrypt registered [ 977.104106] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 978.484150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 978.506226] alg: No test for adler32 (adler32-zlib) [ 979.695741] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 979.988331] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 981.664176] Key type lgssc registered [ 982.713508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 990.492913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 990.524917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.102060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 996.418679] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1001.465295] LustreError: 35016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.499419] LustreError: 35016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1005.194782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.489704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1009.955224] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1017.345247] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1022.722867] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1029.435394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1029.829503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.572504] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1036.994788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1042.692742] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1056.738087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1056.751888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.765180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1057.761814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1057.762955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.790716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1059.434415] Lustre: server umount lustre-OST0000 complete [ 1062.882962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1062.917608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1062.940768] Lustre: Skipped 2 previous similar messages [ 1068.003991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.025504] Lustre: Skipped 1 previous similar message [ 1076.193447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1076.205793] Lustre: Skipped 3 previous similar messages [ 1077.216158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1077.463203] Lustre: server umount lustre-MDT0000 complete [ 1080.763742] LustreError: 34998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380528 with bad export cookie 1024443599342532082 [ 1080.769659] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1080.774826] LustreError: 34998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1081.072439] Lustre: server umount lustre-MDT0001 complete [ 1090.787223] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1093.705598] Key type lgssc unregistered [ 1094.077339] LNet: 38550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.085570] LNetError: 38550:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1094.100742] LNet: Removed LNI 192.168.203.156@tcp [ 1094.872873] Key type .llcrypt unregistered [ 1094.874798] Key type ._llcrypt unregistered [ 1116.580573] Key type ._llcrypt registered [ 1116.582620] Key type .llcrypt registered [ 1116.698595] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1117.743458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1117.991412] alg: No test for adler32 (adler32-zlib) [ 1119.164395] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 1119.415713] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1121.080177] Key type lgssc registered [ 1122.129243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1129.806182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1129.821619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.297476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1135.697233] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1141.736068] LustreError: 39968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1141.752700] LustreError: 39968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1144.792912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1145.241796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1150.327351] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1157.461706] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1163.070839] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1170.309372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1170.888488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1177.081303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1178.416450] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1186.315089] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1192.424504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1192.446963] Lustre: Skipped 1 previous similar message [ 1192.448726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1192.471798] Lustre: Skipped 2 previous similar messages [ 1194.363196] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 1194.754236] Lustre: server umount lustre-MDT0000 complete [ 1197.538952] LustreError: 41207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.577651] LustreError: 41207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1198.701918] LustreError: 39948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380646 with bad export cookie 1812728067132695020 [ 1198.706035] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1198.730041] LustreError: 39948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1199.434782] LustreError: 39965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1199.472395] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.56@tcp (stopping) [ 1202.659557] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.661192] LustreError: 39965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.673121] Lustre: Skipped 1 previous similar message [ 1202.676391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1205.307682] Lustre: server umount lustre-MDT0001 complete [ 1228.045516] Lustre: server umount lustre-OST0000 complete [ 1244.330522] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1246.924895] Key type lgssc unregistered [ 1247.342827] LNet: 43829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.349473] LNetError: 43829:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.364320] LNet: Removed LNI 192.168.203.156@tcp [ 1248.275741] Key type .llcrypt unregistered [ 1248.279991] Key type ._llcrypt unregistered [ 1273.960735] Key type ._llcrypt registered [ 1273.962621] Key type .llcrypt registered [ 1274.066247] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1275.326891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1275.405585] alg: No test for adler32 (adler32-zlib) [ 1276.781495] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 1277.192260] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1279.024185] Key type lgssc registered [ 1280.472388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1288.993913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1289.020037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.548118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.615824] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1300.963403] LustreError: 45248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.983688] LustreError: 45248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1304.533555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1304.966907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1309.870344] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1317.033384] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1322.474606] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1329.562792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1329.793878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1335.947450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1336.178831] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1343.628668] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1354.725300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1354.735337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1354.746387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1354.763205] Lustre: Skipped 1 previous similar message [ 1355.636345] Lustre: lustre-OST0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 1358.767211] Lustre: server umount lustre-OST0000 complete [ 1376.738566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1376.751433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.771135] Lustre: Skipped 1 previous similar message [ 1376.780153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1379.297228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1379.299501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.944298] Lustre: server umount lustre-MDT0000 complete [ 1384.419114] LustreError: 45248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1384.451803] LustreError: 45248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1385.285879] LustreError: 45229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380833 with bad export cookie 8678851994222391494 [ 1385.289972] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1385.300201] LustreError: 45229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1391.657735] Lustre: server umount lustre-MDT0001 complete [ 1403.420922] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1406.041340] Key type lgssc unregistered [ 1406.359487] LNet: 48954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.369640] LNetError: 48954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1407.400643] LNet: Removed LNI 192.168.203.156@tcp [ 1408.233846] Key type .llcrypt unregistered [ 1408.236360] Key type ._llcrypt unregistered [ 1432.267473] Key type ._llcrypt registered [ 1432.270963] Key type .llcrypt registered [ 1432.364186] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1433.355234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1433.379930] alg: No test for adler32 (adler32-zlib) [ 1434.407325] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 1434.601782] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1436.248175] Key type lgssc registered [ 1437.367259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1444.502735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1444.526401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.072061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1451.219469] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1455.592380] LustreError: 50372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.629605] LustreError: 50372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1460.708114] LustreError: 50371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.159843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.665563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1466.524791] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1474.776956] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1480.362043] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1488.290748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1488.782658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1490.896465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1495.335712] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1504.003740] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1513.100281] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1516.837960] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1529.084580] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-22 2h42m53s [ 1530.855718] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1533.410957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1533.420795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1533.448062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1534.498482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1534.504062] Lustre: Skipped 4 previous similar messages [ 1536.885814] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 1539.430976] Lustre: server umount lustre-MDT0000 complete [ 1540.585599] LustreError: 50372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1540.605387] LustreError: 50372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1543.662445] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1544.009525] LustreError: 50367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1550.152825] LustreError: 50368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1550.180514] LustreError: 50368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1552.019344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1552.154282] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1552.431493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.530247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1553.468484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1557.741835] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1561.874378] Lustre: DEBUG MARKER: Wait for df (36562) ... [ 1563.352597] Lustre: DEBUG MARKER: done [ 1568.745108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.747136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1568.766028] Lustre: Skipped 3 previous similar messages [ 1568.780271] Lustre: Skipped 2 previous similar messages [ 1573.858900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1573.866519] Lustre: Skipped 1 previous similar message [ 1574.125431] Lustre: server umount lustre-OST0000 complete [ 1578.981270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.003798] Lustre: Skipped 1 previous similar message [ 1584.114619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1584.127315] Lustre: Skipped 5 previous similar messages [ 1592.288144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1592.441238] Lustre: server umount lustre-MDT0000 complete [ 1596.388321] LustreError: 50372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1596.412846] LustreError: 50372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1596.505268] LustreError: 50350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381044 with bad export cookie 3029442124290354363 [ 1596.513686] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1596.934678] Lustre: server umount lustre-MDT0001 complete [ 1606.747380] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1609.356269] Key type lgssc unregistered [ 1609.632853] LNet: 55295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.640789] LNetError: 55295:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.664219] LNet: Removed LNI 192.168.203.156@tcp [ 1611.462382] Key type .llcrypt unregistered [ 1611.470151] Key type ._llcrypt unregistered [ 1648.819472] Key type ._llcrypt registered [ 1648.825249] Key type .llcrypt registered [ 1648.950283] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1649.781937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1649.869449] alg: No test for adler32 (adler32-zlib) [ 1650.877384] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 1651.086631] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1652.720385] Key type lgssc registered [ 1653.741780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1660.531485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1660.550378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.800435] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1661.841046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1662.002505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.266927] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1674.220520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1674.382111] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1674.408551] Lustre: 57499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1674.691153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1679.116340] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1685.431168] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1690.299119] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1696.752168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1696.983253] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1697.161920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1704.036881] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1706.635608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1711.159210] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1717.016139] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-22 2h46m02s [ 1719.165336] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1726.135436] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1728.622941] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1728.675868] LustreError: 59830:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1748.875755] Lustre: lustre-MDT0000: Client a47d6092-b2d2-4101-9b45-2abbcdb62cf8 (at 192.168.203.56@tcp) reconnecting [ 1768.753377] LustreError: 59830:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1770.938799] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1773.186734] Lustre: DEBUG MARKER: done [ 1780.965800] Lustre: server umount lustre-OST0000 complete [ 1787.363857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1787.373125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.391546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1788.452267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1788.461645] Lustre: Skipped 2 previous similar messages [ 1794.674696] Lustre: server umount lustre-MDT0000 complete [ 1798.015444] LustreError: 57493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381245 with bad export cookie 4501456893894425086 [ 1798.020359] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1798.027582] LustreError: 57493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1807.574447] Lustre: server umount lustre-MDT0001 complete [ 1818.465235] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1822.135663] Key type lgssc unregistered [ 1822.712190] LNet: 61830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.720632] LNetError: 61830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.759323] LNet: Removed LNI 192.168.203.156@tcp [ 1824.229919] Key type .llcrypt unregistered [ 1824.231862] Key type ._llcrypt unregistered [ 1867.334057] Key type ._llcrypt registered [ 1867.341781] Key type .llcrypt registered [ 1867.463777] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1869.012399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1869.059201] alg: No test for adler32 (adler32-zlib) [ 1870.129307] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 1870.381557] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1872.128169] Key type lgssc registered [ 1873.434698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1876.651439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1876.665511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.895580] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1877.922681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1878.132047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1879.053803] Lustre: Failing over lustre-MDT0000 [ 1879.472607] Lustre: server umount lustre-MDT0000 complete [ 1886.779184] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:48:52 (1787381332) [ 1893.761809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.306757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.140921] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1910.140586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.296073] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1910.344949] Lustre: 64634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1910.698167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1915.225918] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1922.762743] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1928.498766] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1935.691737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1935.943733] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1936.154963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.590821] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1943.772834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1950.550573] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1955.313961] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1961.953488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1961.970111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.005601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.521205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.523185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.540034] Lustre: Skipped 1 previous similar message [ 1964.544073] Lustre: Skipped 1 previous similar message [ 1967.801930] Lustre: server umount lustre-MDT0000 complete [ 1971.873560] LustreError: 64626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381419 with bad export cookie 16737453089799276409 [ 1971.886618] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1972.381061] Lustre: server umount lustre-MDT0001 complete [ 1975.472317] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1979.724352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1988.097230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.136846] Lustre: 63556:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787381422/real 1787381422] req@000000007ccba1e6 x1874205152943616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787381438 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1991.170852] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1992.160235] Lustre: 63555:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787381424/real 1787381424] req@0000000013ad08d9 x1874205152943744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787381440 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1995.232215] Lustre: 63558:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787381427/real 1787381427] req@00000000da928bfc x1874205152943872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787381443 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1996.320726] LustreError: 63554:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005d7e1bba x1874205152944640/t0(0) o250->MGC192.168.203.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1996.592107] LustreError: 66483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.606710] LustreError: 66483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1996.672464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1996.763384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1997.667221] Lustre: 63558:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787381429/real 1787381429] req@0000000092dbc3de x1874205152944128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787381445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1999.729758] LustreError: 66484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2000.872289] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2001.763361] LustreError: 66483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.787079] LustreError: 66483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2002.785442] Lustre: 63555:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787381434/real 1787381434] req@00000000095ba29f x1874205152944256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787381450 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2007.013286] LustreError: 66484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.031682] LustreError: 66484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2008.841682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.283358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2014.079805] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2014.700729] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2019.829409] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2021.049633] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2028.159713] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2034.595085] Lustre: server umount lustre-MDT0000 complete [ 2035.171084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2035.182569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.187405] LustreError: 68401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2035.200806] Lustre: Skipped 2 previous similar messages [ 2035.233511] LustreError: 68401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2038.184420] LustreError: 65369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381486 with bad export cookie 16737453089799278684 [ 2038.185484] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2038.198878] LustreError: 65369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2038.632336] Lustre: server umount lustre-MDT0001 complete [ 2045.828744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2056.673215] Lustre: 63557:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787381488/real 1787381488] req@000000006c96116c x1874205152968960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787381504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2056.704521] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.779826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.212213] LustreError: 66483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2064.240205] LustreError: 66483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2064.320437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.382769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2068.792507] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2075.625461] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2077.605605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.954579] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2083.319112] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2089.026799] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2095.801547] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2103.778069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2103.793791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.827443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.837329] Lustre: Skipped 2 previous similar messages [ 2108.901932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2108.910125] Lustre: Skipped 2 previous similar messages [ 2109.857865] Lustre: server umount lustre-MDT0000 complete [ 2114.022102] LustreError: 65369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381561 with bad export cookie 16737453089799280063 [ 2114.039702] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2114.048311] LustreError: 71814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.048322] LustreError: 71814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2114.048511] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2114.051178] LustreError: 65369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2114.270197] Lustre: server umount lustre-MDT0001 complete [ 2121.204606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2124.999144] Lustre: DEBUG MARKER: files compared the same [ 2126.819893] Lustre: server umount lustre-OST0000 complete [ 2143.138582] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 2145.808710] Key type lgssc unregistered [ 2146.077007] LNet: 74354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.083714] LNetError: 74354:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2146.109040] LNet: Removed LNI 192.168.203.156@tcp [ 2146.868928] Key type .llcrypt unregistered [ 2146.873957] Key type ._llcrypt unregistered [ 2169.583190] Key type ._llcrypt registered [ 2169.586112] Key type .llcrypt registered [ 2169.710958] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2170.690336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2170.956921] alg: No test for adler32 (adler32-zlib) [ 2172.019402] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 2172.262568] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 2173.962194] Key type lgssc registered [ 2175.385450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2182.933209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2182.949412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2184.552945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.235893] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug +malloc all [ 2194.920090] LustreError: 75783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.943102] LustreError: 75783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2197.538929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.886270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2201.860744] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug +malloc all [ 2208.793172] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2213.790971] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2220.180741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2220.513273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2226.543292] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug +malloc all [ 2229.742610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2233.077996] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2243.041256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2243.054485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.076034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2245.089150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2245.090795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2245.094704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.124163] Lustre: Skipped 1 previous similar message [ 2246.879520] Lustre: server umount lustre-OST0000 complete [ 2254.305022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2254.324660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2254.350314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2256.811901] Lustre: server umount lustre-MDT0000 complete [ 2260.450589] LustreError: 76511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.475755] LustreError: 76511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2261.056656] LustreError: 75762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381708 with bad export cookie 14629304185568858801 [ 2261.065633] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2261.066052] LustreError: 75762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2261.523361] Lustre: server umount lustre-MDT0001 complete [ 2274.496381] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 2279.218848] Key type lgssc unregistered [ 2279.788349] LNet: 79317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.804464] LNetError: 79317:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2279.848534] LNet: Removed LNI 192.168.203.156@tcp [ 2281.284829] Key type .llcrypt unregistered [ 2281.293616] Key type ._llcrypt unregistered [ 2309.846857] Key type ._llcrypt registered [ 2309.852668] Key type .llcrypt registered [ 2309.942170] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2311.291501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2311.348536] alg: No test for adler32 (adler32-zlib) [ 2312.432780] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 2312.691951] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 2314.336687] Key type lgssc registered [ 2315.600810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2323.854977] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2323.871477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2339.424481] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2344.544326] LustreError: 80707:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.156@tcp: failed processing log, type 4: rc = -110 [ 2371.233345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2377.360444] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2385.162896] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2393.546807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2393.699432] Lustre: *** cfs_fail_loc=706, val=0*** [ 2394.203782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2394.289507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2398.411570] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2399.215492] LustreError: 80733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.232327] LustreError: 80733:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2404.322038] LustreError: 80732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.350599] LustreError: 81794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.361789] LustreError: 81794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2406.650560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2407.107102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2411.407551] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2419.073294] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2424.763272] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2427.879141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2427.897261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2427.921341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2427.938332] Lustre: Skipped 1 previous similar message [ 2433.020268] Lustre: server umount lustre-OST0000 complete [ 2438.115808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2438.124138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2438.133118] Lustre: Skipped 2 previous similar messages [ 2438.148806] Lustre: Skipped 1 previous similar message [ 2442.737977] Lustre: server umount lustre-MDT0000 complete [ 2443.234058] LustreError: 81798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.251259] LustreError: 81798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2446.242875] LustreError: 80713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381894 with bad export cookie 5213866984462095698 [ 2446.256114] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2446.268792] LustreError: 80713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2446.603986] Lustre: server umount lustre-MDT0001 complete [ 2456.688508] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 2459.256906] Key type lgssc unregistered [ 2459.651605] LNet: 84324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2459.660367] LNetError: 84324:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2459.683632] LNet: Removed LNI 192.168.203.156@tcp [ 2460.590328] Key type .llcrypt unregistered [ 2460.599079] Key type ._llcrypt unregistered [ 2485.490725] Key type ._llcrypt registered [ 2485.502530] Key type .llcrypt registered [ 2485.625968] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2487.280942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2487.339063] alg: No test for adler32 (adler32-zlib) [ 2488.570674] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 2488.889411] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 2490.648250] Key type lgssc registered [ 2491.995071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2500.394246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2500.419905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.790500] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2515.485849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2516.086633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2520.185043] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2521.570344] LustreError: 86429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.589121] LustreError: 86429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2525.667308] LustreError: 86433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2525.682520] LustreError: 86433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2528.919515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2529.472548] LustreError: 86429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.511627] LustreError: 86429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2529.544905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.893694] LustreError: 86432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.916131] LustreError: 86432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2536.437362] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2540.006830] LustreError: 86433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.025042] LustreError: 86433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2545.359957] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2545.371462] Lustre: MGS: Client 253cb21e-db60-4b7c-bbc5-579f4ca53e10 (at 0@lo) reconnecting [ 2545.384611] LustreError: 86443:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2545.655861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2545.766583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2550.158371] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2566.116912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2566.123302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2566.133099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2566.138368] Lustre: Skipped 1 previous similar message [ 2569.922837] Lustre: server umount lustre-OST0000 complete [ 2576.354884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.356369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2576.386344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2576.396094] Lustre: Skipped 2 previous similar messages [ 2580.564456] Lustre: server umount lustre-MDT0000 complete [ 2581.478045] LustreError: 86427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.500093] LustreError: 86427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2585.265698] Lustre: server umount lustre-MDT0001 complete [ 2589.434630] Lustre: server umount MGS complete [ 2600.288764] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 03:00:45 (1787382045) [ 2628.387995] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 2639.247506] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2675.189977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.643205] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2689.168391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.347919] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2689.412075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2689.512971] Lustre: lustre-MDT0001: new disk, initializing [ 2689.614073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2689.659826] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2689.670116] Lustre: 93610:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2689.680508] LustreError: 93610:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2693.828413] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2703.699964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2704.014631] Lustre: lustre-OST0000: new disk, initializing [ 2704.028874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2704.040735] Lustre: 94376:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2704.131381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.192563] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2714.146712] Lustre: 94851:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d3c13e1d x1874205800964224/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787382177 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2719.640850] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2725.642764] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2725.657247] Lustre: MGS: Client 7330a7a7-1d35-463b-8068-14fe36780a4d (at 0@lo) reconnecting [ 2725.685704] Lustre: 92880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2725.697315] LustreError: 93596:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2725.724815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2725.786714] Lustre: lustre-MDT0000: new disk, initializing [ 2725.937186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2725.965959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2726.094426] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2727.009477] Lustre: 95444:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d72bf578 x1874205800970752/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787382190 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2730.045201] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2733.544795] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2736.110064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2736.119197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2736.268573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2759.650936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2759.659213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2759.673749] Lustre: Skipped 2 previous similar messages [ 2759.684684] Lustre: Skipped 1 previous similar message [ 2764.772334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2764.783459] Lustre: Skipped 1 previous similar message [ 2764.925761] Lustre: server umount lustre-OST0000 complete [ 2769.894091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2769.901624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2769.912375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2769.914942] Lustre: Skipped 1 previous similar message [ 2769.937880] Lustre: Skipped 1 previous similar message [ 2775.368949] Lustre: server umount lustre-MDT0000 complete [ 2780.131177] LustreError: 93580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.141872] LustreError: 93580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2786.713319] Lustre: server umount lustre-MDT0001 complete [ 2791.303388] Lustre: server umount MGS complete [ 2802.197386] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 03:04:07 (1787382247) [ 2818.572634] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 2821.070784] Key type lgssc unregistered [ 2821.375859] LNet: 97944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2821.381751] LNetError: 97944:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2821.404593] LNet: Removed LNI 192.168.203.156@tcp [ 2822.205151] Key type .llcrypt unregistered [ 2822.211255] Key type ._llcrypt unregistered [ 2838.108381] Key type ._llcrypt registered [ 2838.110372] Key type .llcrypt registered [ 2838.204196] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2839.384975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2839.427508] alg: No test for adler32 (adler32-zlib) [ 2840.500697] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 2840.728530] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 2842.402091] Key type lgssc registered [ 2843.537795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2850.255857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2850.292611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.757887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2856.340554] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2862.060641] LustreError: 99167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.092271] LustreError: 99167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2864.679605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.207313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2870.152353] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2877.688882] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2883.737828] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2890.006708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2890.369355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.241540] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2896.512897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2903.260370] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2915.299863] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2915.305443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.312124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2915.317863] Lustre: Skipped 1 previous similar message [ 2915.333793] Lustre: Skipped 1 previous similar message [ 2916.562874] Lustre: server umount lustre-OST0000 complete [ 2920.422612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.428469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2920.434538] Lustre: Skipped 1 previous similar message [ 2920.451280] Lustre: Skipped 1 previous similar message [ 2925.544776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2925.556750] Lustre: Skipped 1 previous similar message [ 2930.657519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2930.665162] Lustre: Skipped 1 previous similar message [ 2934.240337] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2934.535768] Lustre: server umount lustre-MDT0000 complete [ 2935.781357] LustreError: 99168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2935.796112] LustreError: 99168:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2937.726312] LustreError: 99149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382385 with bad export cookie 5098602393563899682 [ 2937.732787] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2937.736418] LustreError: 99149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2938.034520] Lustre: server umount lustre-MDT0001 complete [ 2947.938289] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 2950.851708] Key type lgssc unregistered [ 2951.233554] LNet: 102701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2951.245273] LNetError: 102701:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2952.294646] LNet: Removed LNI 192.168.203.156@tcp [ 2953.144118] Key type .llcrypt unregistered [ 2953.146222] Key type ._llcrypt unregistered [ 2969.542432] Key type ._llcrypt registered [ 2969.544631] Key type .llcrypt registered [ 2969.646400] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2970.733113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2970.819116] alg: No test for adler32 (adler32-zlib) [ 2971.892992] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 2972.194515] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 2973.904369] Key type lgssc registered [ 2975.179097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2981.694470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2981.712037] LustreError: 103782:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2981.746526] LustreError: 103786:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2981.757461] LustreError: 103782:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4960 [ 2981.768554] LustreError: 103782:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2981.776817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2981.782445] LustreError: 103782:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2983.242175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2989.889441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.194375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2994.052948] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2999.808885] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing lsmod [ 3003.275218] LustreError: 104992:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3003.280911] LustreError: 104993:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3003.286446] LustreError: 104992:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 3003.291903] LustreError: 104993:0:(obd_mount.c:234:lustre_start_simple()) lustre-OST0000-osd attach error -17 [ 3003.304874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3003.307699] LustreError: 104993:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -17 [ 3003.320696] LustreError: 104993:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 3003.830542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.997631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.011485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3009.023287] Lustre: Skipped 2 previous similar messages [ 3009.049341] Lustre: Skipped 2 previous similar messages [ 3012.068427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3012.089843] Lustre: Skipped 2 previous similar messages [ 3013.572351] Lustre: server umount lustre-MDT0000 complete [ 3015.144529] LustreError: 105178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3015.166275] LustreError: 105178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3017.370897] LustreError: 103850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382465 with bad export cookie 17897571550581988229 [ 3017.382262] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3017.386015] LustreError: 103850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3019.361901] LustreError: 104256:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d5f2b9c7 x1874206308454016/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3019.601272] Lustre: server umount lustre-MDT0001 complete [ 3034.114332] Lustre: server umount lustre-OST0000 complete [ 3042.320345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3042.734991] LustreError: 106054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.843750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.669777] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3047.915833] LustreError: 106055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.026798] LustreError: 106054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.143265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.452887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3058.674591] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3065.895831] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3071.702428] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3079.412375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3079.842776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.681630] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3088.076668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3093.142405] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3101.666845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3101.676044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.703569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3103.716594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3103.720839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3103.724228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.743046] Lustre: Skipped 1 previous similar message [ 3106.113842] Lustre: server umount lustre-OST0000 complete [ 3110.883079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3110.891628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.902728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3116.368014] Lustre: server umount lustre-MDT0000 complete [ 3119.073252] LustreError: 106054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.106526] LustreError: 106054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3120.199522] LustreError: 106036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382568 with bad export cookie 17897571550581988922 [ 3120.203265] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3120.219326] LustreError: 106036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3120.618147] Lustre: server umount lustre-MDT0001 complete [ 3132.239400] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3134.827511] Key type lgssc unregistered [ 3135.143820] LNet: 109589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3135.150871] LNetError: 109589:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3135.164845] LNet: Removed LNI 192.168.203.156@tcp [ 3136.063464] Key type .llcrypt unregistered [ 3136.065492] Key type ._llcrypt unregistered [ 3161.513462] Key type ._llcrypt registered [ 3161.519880] Key type .llcrypt registered [ 3161.629254] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3162.523941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3162.571346] alg: No test for adler32 (adler32-zlib) [ 3163.676412] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 3163.914749] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3165.625344] Key type lgssc registered [ 3166.692914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3174.648415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3174.678441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3176.209814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3180.572158] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3186.661137] LustreError: 111009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3186.687058] LustreError: 111009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3188.697632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3189.184423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3193.941857] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3201.046749] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3206.866852] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3213.762721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3214.060414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.454701] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3221.231741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3225.365705] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3238.646526] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3241.362211] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3246.554529] Lustre: server umount lustre-OST0000 complete [ 3250.657559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3250.666915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.684087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3254.244063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3254.246747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.366738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.374441] Lustre: Skipped 2 previous similar messages [ 3260.495415] Lustre: server umount lustre-MDT0000 complete [ 3264.448371] LustreError: 110989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382712 with bad export cookie 12345973178862662794 [ 3264.450882] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3264.462965] LustreError: 110989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3264.469604] Lustre: 110319:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787382696/real 1787382696] req@00000000912a1ead x1874206508747008/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787382712 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3264.469657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.826561] Lustre: server umount lustre-MDT0001 complete [ 3275.488512] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3278.366557] Key type lgssc unregistered [ 3278.689125] LNet: 114668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3278.701426] LNetError: 114668:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3278.725158] LNet: Removed LNI 192.168.203.156@tcp [ 3279.572861] Key type .llcrypt unregistered [ 3279.576278] Key type ._llcrypt unregistered [ 3295.501741] Key type ._llcrypt registered [ 3295.505923] Key type .llcrypt registered [ 3295.615614] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3296.794314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3296.856969] alg: No test for adler32 (adler32-zlib) [ 3297.918321] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 3298.198500] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3299.849595] Key type lgssc registered [ 3300.911769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3307.869240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3307.893398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3309.285200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3313.416138] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3319.788076] LustreError: 115887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.798407] LustreError: 115887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3322.049900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3322.456632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3326.718718] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3333.749530] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3339.951996] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3346.793176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3347.177618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3353.056080] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3354.351256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3360.001335] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3368.930789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3368.947625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.961042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3369.955695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3369.965369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.969772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3369.995552] Lustre: Skipped 1 previous similar message [ 3373.765580] Lustre: server umount lustre-OST0000 complete [ 3378.660277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3378.669713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3378.684820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3383.406755] Lustre: server umount lustre-MDT0000 complete [ 3385.313983] LustreError: 115881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.338496] LustreError: 115881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3386.667126] LustreError: 115869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382834 with bad export cookie 163411000430463514 [ 3386.668588] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3386.682791] LustreError: 115869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3387.020713] Lustre: server umount lustre-MDT0001 complete [ 3398.798474] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3401.473297] Key type lgssc unregistered [ 3401.749441] LNet: 119423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3401.758523] LNetError: 119423:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3401.792972] LNet: Removed LNI 192.168.203.156@tcp [ 3402.761085] Key type .llcrypt unregistered [ 3402.764189] Key type ._llcrypt unregistered [ 3427.030036] Key type ._llcrypt registered [ 3427.033225] Key type .llcrypt registered [ 3427.141701] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3429.008432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3429.046780] alg: No test for adler32 (adler32-zlib) [ 3430.271296] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 3430.761214] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3432.571038] Key type lgssc registered [ 3434.303233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3441.605992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3441.629075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3443.073734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3447.897900] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3452.386908] LustreError: 120840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.406954] LustreError: 120840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3454.485853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.867669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3458.820805] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3466.343508] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3471.506526] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3478.796881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3479.235961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.759714] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3487.408775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3494.860436] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3500.673387] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3503.991983] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3504.002522] Lustre: Skipped 2 previous similar messages [ 3508.103266] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3508.112950] Lustre: Skipped 2 previous similar messages [ 3509.882769] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3509.920486] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3519.039137] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.56@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3519.047604] Lustre: Skipped 2 previous similar messages [ 3525.673416] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.56@tcp 192.168.0.[2,10]@tcp [ 3530.165421] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.56/24@tcp in log lustre-MDT0000 [ 3530.170388] Lustre: Skipped 2 previous similar messages [ 3534.123776] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.56/24@tcp [ 3534.127715] Lustre: Skipped 1 previous similar message [ 3541.992971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3542.004972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.017483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3543.017795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3543.019845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3543.029404] LustreError: Skipped 2 previous similar messages [ 3543.058066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3548.214607] Lustre: server umount lustre-OST0000 complete [ 3552.225451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3552.232812] LustreError: Skipped 2 previous similar messages [ 3552.238355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3552.254947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3552.258866] Lustre: Skipped 2 previous similar messages [ 3554.276928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3554.285326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3554.302576] Lustre: Skipped 1 previous similar message [ 3557.922980] Lustre: server umount lustre-MDT0000 complete [ 3559.398879] LustreError: 123491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3559.422556] LustreError: 123491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3561.193258] LustreError: 123471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383009 with bad export cookie 10388055369932977127 [ 3561.202267] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3561.212849] LustreError: 123471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3561.549501] Lustre: server umount lustre-MDT0001 complete [ 3572.558853] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3575.314826] Key type lgssc unregistered [ 3575.536706] LNet: 125730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3575.542279] LNetError: 125730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3575.552457] LNet: Removed LNI 192.168.203.156@tcp [ 3576.196391] Key type .llcrypt unregistered [ 3576.198102] Key type ._llcrypt unregistered [ 3601.499783] Key type ._llcrypt registered [ 3601.503701] Key type .llcrypt registered [ 3601.599849] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3602.408115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3602.526469] alg: No test for adler32 (adler32-zlib) [ 3603.622937] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 3603.874544] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3605.584167] Key type lgssc registered [ 3606.810984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3614.582802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3614.618237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3616.189767] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3616.206388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3616.218608] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.56/24@tcp [ 3620.846700] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3625.444079] LustreError: 127288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3625.454476] LustreError: 127288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3629.529209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3630.054852] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3630.099931] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.56/24@tcp [ 3630.168045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3634.693423] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3641.649683] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3647.059651] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3653.518326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3653.919478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.927441] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3661.132679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3666.578504] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3676.642323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3676.642604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3676.651066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3676.664836] Lustre: Skipped 1 previous similar message [ 3681.762545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3681.776569] Lustre: Skipped 2 previous similar messages [ 3682.753688] Lustre: server umount lustre-OST0000 complete [ 3686.370663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3686.376629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3686.393336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3692.004380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3692.006362] Lustre: Skipped 2 previous similar messages [ 3692.486502] Lustre: server umount lustre-MDT0000 complete [ 3697.126667] LustreError: 127288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3697.179478] LustreError: 127288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3697.532948] LustreError: 127270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383145 with bad export cookie 14393990565607135171 [ 3697.546507] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3697.552980] LustreError: 127270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3698.162920] Lustre: server umount lustre-MDT0001 complete [ 3709.245528] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3712.221836] Key type lgssc unregistered [ 3712.477727] LNet: 130876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3712.484954] LNetError: 130876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3712.499776] LNet: Removed LNI 192.168.203.156@tcp [ 3713.175259] Key type .llcrypt unregistered [ 3713.182305] Key type ._llcrypt unregistered [ 3736.986423] Key type ._llcrypt registered [ 3736.992258] Key type .llcrypt registered [ 3737.080691] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 3749.535314] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3750.287726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3750.376071] alg: No test for adler32 (adler32-zlib) [ 3751.396765] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 3751.606188] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3753.240162] Key type lgssc registered [ 3754.474853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3788.947253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3788.982896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3790.162527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3790.183461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3790.242865] Lustre: lustre-MDT0000: new disk, initializing [ 3790.344459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3790.390576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3795.032059] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3805.732628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3805.866333] Lustre: 134255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3805.904526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3805.908512] Lustre: Skipped 1 previous similar message [ 3805.999904] Lustre: lustre-MDT0001: new disk, initializing [ 3806.125045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3806.200645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3806.221514] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3811.150444] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3820.954809] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3827.055218] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3833.660992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3834.035393] Lustre: lustre-OST0000: new disk, initializing [ 3834.045777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3834.053598] Lustre: 136223:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3834.178389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3840.081129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3840.090346] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3840.095706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3840.232030] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3849.051895] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3853.392597] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3853.568289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3857.771136] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3858.066974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3860.963149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3860.968470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.973171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3860.993132] Lustre: Skipped 1 previous similar message [ 3861.033919] Lustre: Skipped 1 previous similar message [ 3866.093412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3866.099560] Lustre: Skipped 1 previous similar message [ 3871.205537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3871.211847] Lustre: Skipped 1 previous similar message [ 3874.272199] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3874.449310] Lustre: server umount lustre-OST0000 complete [ 3881.446975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3881.461544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3881.471882] Lustre: Skipped 1 previous similar message [ 3884.248909] Lustre: server umount lustre-MDT0000 complete [ 3887.517548] LustreError: 134247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383335 with bad export cookie 3672034393505834568 [ 3887.525328] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3887.526181] LustreError: 134247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3888.000809] Lustre: server umount lustre-MDT0001 complete [ 3898.439546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3899.313539] LustreError: 138454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.352698] LustreError: 138454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3899.493114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3904.491407] LustreError: 138455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3905.032474] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3909.608181] LustreError: 138454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3914.223040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.648462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3918.934508] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3927.202413] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3932.460775] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3940.489109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3940.908944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3947.543706] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3954.988443] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3958.956456] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3959.212234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3963.720349] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3963.998508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3973.181614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3973.386602] Lustre: lustre-OST0001: new disk, initializing [ 3973.390292] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3973.399403] Lustre: 141826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3973.536883] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3980.087469] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3980.406566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3980.416928] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3980.558957] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3988.425619] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3988.592283] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3992.875769] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3993.116501] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3997.972674] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9aeb473b4000.ost_server_uuid 50 [ 3999.318783] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9aeb473b4000.ost_server_uuid in FULL state after 0 sec [ 4009.210645] Lustre: server umount lustre-OST0001 complete [ 4011.490137] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4011.493899] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.505105] LustreError: Skipped 2 previous similar messages [ 4011.527439] Lustre: Skipped 1 previous similar message [ 4011.533313] LustreError: 140275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4011.564840] LustreError: 140275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4015.586035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4015.594422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.615281] Lustre: Skipped 1 previous similar message [ 4015.627803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4016.610843] LustreError: 140275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4016.640315] LustreError: 140275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4026.859907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4026.860157] LustreError: 140277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4026.878867] Lustre: Skipped 5 previous similar messages [ 4026.914899] LustreError: 140277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4027.360140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4027.622607] Lustre: server umount lustre-OST0000 complete [ 4031.970813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4031.993325] Lustre: Skipped 2 previous similar messages [ 4037.682518] Lustre: server umount lustre-MDT0000 complete [ 4040.813023] LustreError: 138435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383488 with bad export cookie 3672034393505835457 [ 4040.824817] LustreError: 138435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4040.841933] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4041.105084] Lustre: server umount lustre-MDT0001 complete [ 4059.100973] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 4062.642969] Key type lgssc unregistered [ 4062.977617] LNet: 144829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4062.994189] LNetError: 144829:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4063.013155] LNet: Removed LNI 192.168.203.156@tcp [ 4064.198648] Key type .llcrypt unregistered [ 4064.201713] Key type ._llcrypt unregistered [ 4106.548674] Key type ._llcrypt registered [ 4106.552389] Key type .llcrypt registered [ 4106.798564] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 4124.236146] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4125.692483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4125.728323] alg: No test for adler32 (adler32-zlib) [ 4126.888356] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 4127.145820] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 4128.912867] Key type lgssc registered [ 4130.307102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4164.049813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4164.077947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4165.436506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4165.475403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4165.553363] Lustre: lustre-MDT0000: new disk, initializing [ 4165.622734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4165.637509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4170.047633] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4181.488472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4181.572405] Lustre: 148983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4181.635753] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4181.644237] Lustre: Skipped 1 previous similar message [ 4181.717340] Lustre: lustre-MDT0001: new disk, initializing [ 4181.842149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4181.902061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4181.912737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4186.346561] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4195.554715] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4201.088730] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4207.360373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4207.634369] Lustre: lustre-OST0000: new disk, initializing [ 4207.637983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4207.653539] Lustre: 150950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4207.786113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4214.670769] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4214.837165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4214.857698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4214.963584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4227.101199] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4233.747272] Lustre: Failing over lustre-OST0000 [ 4233.927435] Lustre: server umount lustre-OST0000 complete [ 4251.104330] Lustre: 147206:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383682/real 1787383682] req@000000000ba8f519 x1874207518525184/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787383698 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4251.158352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.192705] Lustre: 147207:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383683/real 1787383683] req@0000000005ee64a5 x1874207518525312/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787383699 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4252.219473] Lustre: 147207:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4252.237701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.836243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4253.194175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4253.210799] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4256.267266] Lustre: 147206:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383688/real 1787383688] req@0000000083dff29d x1874207518527232/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787383704 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4256.296756] Lustre: 147206:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4258.791988] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4261.191655] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4261.344124] Lustre: 147207:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383693/real 1787383693] req@00000000fc055de8 x1874207518528256/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787383709 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4261.375318] Lustre: 147207:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4262.254766] Lustre: Failing over lustre-MDT0000 [ 4262.265543] LustreError: 147205:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4262.491313] Lustre: server umount lustre-MDT0000 complete [ 4263.337897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.345529] Lustre: Skipped 2 previous similar messages [ 4263.367475] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4263.367760] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4267.488255] Lustre: 147207:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383699/real 1787383699] req@000000000f7cb593 x1874207518529408/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787383715 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4267.523511] Lustre: 147207:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4268.516209] LustreError: 148989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4268.537597] LustreError: 148989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4271.440404] LustreError: 151477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4273.634435] LustreError: 148988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4273.671117] LustreError: 148988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4276.552429] LustreError: 148989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4279.780353] Lustre: 147208:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383711/real 1787383711] req@00000000817e32a5 x1874207518538624/t0(0) o400->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787383727 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4279.801554] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4280.188243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4281.679380] LustreError: 148988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4281.713082] LustreError: 148988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4290.017882] LustreError: 148989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.019071] LustreError: 147205:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b9f111c4 x1874207518545024/t0(0) o250->MGC192.168.203.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4290.039679] LustreError: 148989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4290.258551] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4290.319368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4290.382731] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4291.913126] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4294.599910] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4295.679839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4295.706606] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4295.755956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4304.359626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4304.370780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4304.396679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4305.891314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4308.303602] Lustre: server umount lustre-OST0000 complete [ 4316.129854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4316.132296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.140325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4316.140680] Lustre: Skipped 1 previous similar message [ 4316.168800] Lustre: Skipped 2 previous similar messages [ 4318.715933] Lustre: server umount lustre-MDT0000 complete [ 4321.249731] LustreError: 148989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4321.272184] LustreError: 148989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4322.754127] LustreError: 148976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383770 with bad export cookie 10953674885375149879 [ 4322.763893] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4323.338665] Lustre: server umount lustre-MDT0001 complete [ 4334.258658] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 4337.248635] Key type lgssc unregistered [ 4337.529663] LNet: 154610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4337.537241] LNetError: 154610:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4338.598758] LNet: Removed LNI 192.168.203.156@tcp [ 4339.415781] Key type .llcrypt unregistered [ 4339.418337] Key type ._llcrypt unregistered [ 4377.142555] Key type ._llcrypt registered [ 4377.145197] Key type .llcrypt registered [ 4377.324757] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4379.134010] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4379.377513] alg: No test for adler32 (adler32-zlib) [ 4380.484661] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 4380.977967] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 4382.836550] Key type lgssc registered [ 4384.219547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4393.631565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4393.652022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4395.296791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.468857] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4405.730494] LustreError: 156132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4405.749961] LustreError: 156132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4409.424861] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4410.852338] LustreError: 156133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.632374] LustreError: 156161:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4413.643622] LustreError: 156161:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4413.650144] LustreError: 156161:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4413.985884] Lustre: server umount lustre-MDT0000 complete [ 4432.061635] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 4441.883300] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4476.816673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4477.067816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4477.131359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4477.260000] Lustre: lustre-MDT0000: new disk, initializing [ 4477.345568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4477.375716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4481.670684] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4491.849946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4491.971164] Lustre: 159464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4492.020548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4492.023768] Lustre: Skipped 1 previous similar message [ 4492.084930] Lustre: lustre-MDT0001: new disk, initializing [ 4492.159515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4492.212050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4492.229067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4496.533487] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4505.691904] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4512.010216] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4519.712603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4520.057232] Lustre: lustre-OST0000: new disk, initializing [ 4520.061899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4520.074045] Lustre: 161432:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4520.212160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4528.183760] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4530.247154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4530.256991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4530.373509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4538.292289] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4542.049236] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4544.122830] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:33:09 (1787383989) [ 4789.217918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4789.225207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4789.249896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4791.778102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4791.780639] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4795.412244] Lustre: server umount lustre-MDT0000 complete [ 4796.899736] LustreError: 163218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4796.929976] LustreError: 163218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4798.862062] LustreError: 159455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384246 with bad export cookie 10035452201822674799 [ 4798.873697] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4798.879129] LustreError: 159455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4802.019596] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4802.020426] LustreError: 159471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4802.036407] Lustre: Skipped 1 previous similar message [ 4802.057199] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4802.063977] Lustre: Skipped 2 previous similar messages [ 4808.160960] LustreError: 159471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4808.164568] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4808.184744] LustreError: 159471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4808.193693] Lustre: Skipped 1 previous similar message [ 4813.281081] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4813.282731] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 4813.527270] Lustre: server umount lustre-MDT0001 complete [ 4823.762305] Lustre: server umount lustre-OST0000 complete [ 4835.211666] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 4844.873149] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4881.779156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4882.134523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4882.163991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4882.282715] Lustre: lustre-MDT0000: new disk, initializing [ 4882.390993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4882.418127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4887.099593] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4898.157810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4898.338868] Lustre: 166093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4898.439752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4898.442897] Lustre: Skipped 1 previous similar message [ 4898.560796] Lustre: lustre-MDT0001: new disk, initializing [ 4898.676562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4898.736529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4898.746211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4903.757873] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4913.082763] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4919.310738] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4927.186914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4927.516305] Lustre: lustre-OST0000: new disk, initializing [ 4927.519861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4927.546113] Lustre: 168064:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4927.618891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4934.739471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4934.756277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4934.784947] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4934.916437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4944.767874] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4949.490452] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4949.691447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4954.779476] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4955.009473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4960.740381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4960.743592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4960.753047] LustreError: Skipped 1 previous similar message [ 4960.762599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4960.780605] Lustre: Skipped 1 previous similar message [ 4960.812576] Lustre: Skipped 1 previous similar message [ 4963.377089] Lustre: server umount lustre-OST0000 complete [ 4970.466541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4970.479963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4973.628948] Lustre: server umount lustre-MDT0000 complete [ 4976.102308] LustreError: 166106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.128486] LustreError: 166106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4977.727609] LustreError: 166086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384425 with bad export cookie 10035452201822833629 [ 4977.729950] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4977.753450] LustreError: 166086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4978.185766] Lustre: server umount lustre-MDT0001 complete [ 4987.776150] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:40:33 (1787384433) [ 5004.243333] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 5014.844110] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5053.937334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5054.177766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5054.208848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5054.284716] Lustre: lustre-MDT0000: new disk, initializing [ 5054.388186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5054.418910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5059.150935] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5070.752152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.916319] Lustre: 172824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5071.289095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5076.749405] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5087.506471] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5093.094775] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5100.668167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5100.991404] Lustre: lustre-OST0000: new disk, initializing [ 5100.993917] Lustre: Skipped 1 previous similar message [ 5100.997320] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5101.002905] Lustre: Skipped 2 previous similar messages [ 5101.011203] Lustre: 174800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5107.298671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5107.313606] Lustre: Skipped 1 previous similar message [ 5107.321197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5107.452733] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5107.742433] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5117.823750] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5128.162933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5128.165828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5128.174161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5128.174169] Lustre: Skipped 2 previous similar messages [ 5128.196142] LustreError: Skipped 1 previous similar message [ 5128.260742] Lustre: Skipped 2 previous similar messages [ 5130.783654] Lustre: server umount lustre-OST0000 complete [ 5137.892400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5141.249430] Lustre: server umount lustre-MDT0000 complete [ 5143.525393] LustreError: 172830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5143.552504] LustreError: 172830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5145.255660] LustreError: 172814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384593 with bad export cookie 10035452201822834511 [ 5145.258081] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5145.272559] LustreError: 172814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5145.855573] Lustre: server umount lustre-MDT0001 complete [ 5155.220135] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:43:20 (1787384600) [ 5170.816823] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 5179.221299] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5214.615740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.837823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5214.862555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5214.951231] Lustre: lustre-MDT0000: new disk, initializing [ 5215.080828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5215.087550] Lustre: Skipped 2 previous similar messages [ 5215.119539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5219.529040] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5231.226234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5231.431432] Lustre: 179123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5231.892490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5237.262780] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5247.563854] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5255.462696] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5262.915231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5263.219390] Lustre: 181104:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5269.722919] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5271.142581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5271.238652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5279.204494] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5286.884492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5286.885680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5286.888696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5286.888704] Lustre: Skipped 4 previous similar messages [ 5286.889740] LustreError: Skipped 1 previous similar message [ 5286.916493] Lustre: Skipped 3 previous similar messages [ 5292.604615] Lustre: server umount lustre-OST0000 complete [ 5302.660996] Lustre: server umount lustre-MDT0000 complete [ 5305.317397] LustreError: 179131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5305.339563] LustreError: 179131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5306.501728] LustreError: 181103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384754 with bad export cookie 10035452201822835596 [ 5306.508628] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5306.515791] LustreError: 181103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5306.870559] Lustre: server umount lustre-MDT0001 complete [ 5318.217927] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 5321.154858] Key type lgssc unregistered [ 5321.557232] LNet: 182959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5321.564742] LNetError: 182959:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5321.589224] LNet: Removed LNI 192.168.203.156@tcp [ 5322.623953] Key type .llcrypt unregistered [ 5322.629863] Key type ._llcrypt unregistered [ 5348.058439] Key type ._llcrypt registered [ 5348.061792] Key type .llcrypt registered [ 5348.201521] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5349.320988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5349.371541] alg: No test for adler32 (adler32-zlib) [ 5350.716758] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 5351.066480] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 5352.848183] Key type lgssc registered [ 5353.830372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5362.537779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5362.557154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5364.042195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5369.115244] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5373.416871] LustreError: 184374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.448996] LustreError: 184374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5378.521964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5378.542388] LustreError: 184375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5378.996591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5383.295739] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5390.172945] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5395.686916] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5402.349677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5402.677308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5408.886885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5409.098739] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5416.927587] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5434.338730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5434.343281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5434.352701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5434.852090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5434.864979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5434.892285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5439.995483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5440.008413] Lustre: Skipped 1 previous similar message [ 5440.691262] Lustre: server umount lustre-OST0000 complete [ 5445.601257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5445.611619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5445.638874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5450.214467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5450.218415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.517855] Lustre: server umount lustre-MDT0000 complete [ 5454.561242] LustreError: 184356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384902 with bad export cookie 4377479093664638141 [ 5454.572700] LustreError: 184356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5454.576025] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5454.860449] Lustre: server umount lustre-MDT0001 complete [ 5466.150350] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 5469.031947] Key type lgssc unregistered [ 5469.276969] LNet: 187933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5469.288566] LNetError: 187933:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5469.314641] LNet: Removed LNI 192.168.203.156@tcp [ 5470.136556] Key type .llcrypt unregistered [ 5470.143322] Key type ._llcrypt unregistered [ 5494.471899] Key type ._llcrypt registered [ 5494.474315] Key type .llcrypt registered [ 5494.584407] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5495.702953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5495.782972] alg: No test for adler32 (adler32-zlib) [ 5496.945847] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 5497.331565] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 5499.056571] Key type lgssc registered [ 5500.441816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5508.123537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5508.139654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5509.625401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5513.637554] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5519.845322] LustreError: 189351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5519.860688] LustreError: 189351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5522.043240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.425064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5526.743166] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5533.881294] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5538.979577] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5545.572761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5546.066741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5553.523034] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5554.301110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5561.563916] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5568.365196] Lustre: lustre-OST0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 5568.993846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5569.001360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.013867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5570.021134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5570.023829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5570.028564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5573.448622] Lustre: lustre-OST0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 5573.458556] Lustre: Skipped 1 previous similar message [ 5578.580424] Lustre: lustre-OST0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 5578.602124] Lustre: Skipped 2 previous similar messages [ 5581.285942] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5581.517600] Lustre: server umount lustre-OST0000 complete [ 5589.858293] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9aeb5868e800.ost_server_uuid 50 [ 5591.628986] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5609.442177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5609.464181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5609.482467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5609.492555] Lustre: Skipped 2 previous similar messages [ 5612.521264] Lustre: server umount lustre-MDT0000 complete [ 5615.594335] LustreError: 189350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5615.629304] LustreError: 189350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5617.087487] LustreError: 189332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385064 with bad export cookie 10495418443740285023 [ 5617.098115] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5617.113323] LustreError: 189332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5617.411530] Lustre: server umount lustre-MDT0001 complete [ 5627.161343] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:51:12 (1787385072) [ 5635.978220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.613622] LustreError: 193268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5636.671644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5641.573945] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5641.705312] LustreError: 193269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5646.822202] LustreError: 193268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5650.437479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5650.738858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5655.330290] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5662.519894] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5669.159532] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5676.987923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5677.342685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.461221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5683.722400] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5691.044639] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5698.044556] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5698.312563] Lustre: lustre-OST0001: new disk, initializing [ 5698.316188] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5698.327461] Lustre: 196111:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5698.456602] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5705.070707] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5708.343726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5708.357481] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5708.471627] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5713.868884] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5717.482378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5717.488458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5717.512779] Lustre: Skipped 1 previous similar message [ 5717.525854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5717.534736] Lustre: Skipped 3 previous similar messages [ 5722.662502] Lustre: server umount lustre-OST0000 complete [ 5724.154691] LustreError: 196028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5724.181421] LustreError: 196028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5729.520841] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5729.745404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5734.294536] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5734.616823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5740.376381] LustreError: 195092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5740.407849] LustreError: 195092:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5749.732467] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5749.737206] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5749.756747] Lustre: Skipped 1 previous similar message [ 5749.764437] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5749.775754] LustreError: Skipped 1 previous similar message [ 5749.800294] Lustre: Skipped 4 previous similar messages [ 5754.822839] Lustre: server umount lustre-OST0001 complete [ 5759.969977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.997707] Lustre: Skipped 2 previous similar messages [ 5765.566551] Lustre: server umount lustre-MDT0000 complete [ 5770.669903] LustreError: 193250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385218 with bad export cookie 10495418443740286101 [ 5770.687025] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5770.694167] LustreError: 193250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5771.296098] Lustre: server umount lustre-MDT0001 complete [ 5798.724197] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:54:03 (1787385243) [ 5805.606853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5805.972568] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5806.028291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5806.218545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.480928] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5820.515043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5820.713640] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5820.801744] Lustre: 199552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5826.086824] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5836.347740] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5844.000235] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5852.431759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5852.745044] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5860.191509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5860.834890] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5870.959301] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5879.069067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5879.134087] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5879.374549] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5879.382059] Lustre: Skipped 2 previous similar messages [ 5886.090684] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5887.212175] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5894.732418] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5900.658495] Lustre: lustre-OST0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 5900.666959] Lustre: Skipped 4 previous similar messages [ 5901.285328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5901.286976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5901.303077] LustreError: Skipped 1 previous similar message [ 5904.384947] Lustre: server umount lustre-OST0000 complete [ 5905.749757] LustreError: 203055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5905.787463] LustreError: 203055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5923.312987] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5928.370546] Lustre: server umount lustre-OST0001 complete [ 5933.542019] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5933.564590] Lustre: Skipped 3 previous similar messages [ 5939.023501] Lustre: server umount lustre-MDT0000 complete [ 5943.053324] LustreError: 199544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385390 with bad export cookie 10495418443740287508 [ 5943.069901] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5943.071675] LustreError: 199544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5943.595789] Lustre: server umount lustre-MDT0001 complete [ 5970.652944] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:56:55 (1787385415) [ 5990.104534] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 6001.610724] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 6038.599601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6038.821725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6038.873542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6038.992609] Lustre: lustre-MDT0000: new disk, initializing [ 6039.069579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6039.086547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6043.828906] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6054.022363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6054.173183] Lustre: 207647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6054.218852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6054.225292] Lustre: Skipped 1 previous similar message [ 6054.321257] Lustre: lustre-MDT0001: new disk, initializing [ 6054.520879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6054.535540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6058.975512] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6070.023334] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6076.305665] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6083.799196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6084.270964] Lustre: lustre-OST0000: new disk, initializing [ 6084.278254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6084.284589] Lustre: 209617:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6090.871807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6090.876559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6091.045973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6091.857980] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6102.139441] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6107.772742] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6108.069780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6114.138477] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6114.486161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6121.954824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6121.958065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.966778] LustreError: Skipped 2 previous similar messages [ 6121.981498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6121.988622] Lustre: Skipped 2 previous similar messages [ 6122.012644] Lustre: Skipped 11 previous similar messages [ 6123.500547] Lustre: server umount lustre-OST0000 complete [ 6134.002610] Lustre: server umount lustre-MDT0000 complete [ 6137.315964] LustreError: 207655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6137.351481] LustreError: 207655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 6138.177353] LustreError: 207640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385586 with bad export cookie 10495418443740289293 [ 6138.204574] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6138.220808] LustreError: 207640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6138.813739] Lustre: server umount lustre-MDT0001 complete [ 6149.825864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6156.330854] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6165.610472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6171.073322] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6178.998625] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6185.485628] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6193.747472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6201.199076] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6209.746577] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6215.479442] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6215.791177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6220.085092] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6220.344861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6228.596579] Lustre: server umount lustre-OST0000 complete [ 6236.591633] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6241.597451] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6271.212513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6278.491942] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6286.027337] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6288.924911] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6291.119517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.142610] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.152356] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6291.169806] Lustre: Skipped 1 previous similar message [ 6296.547245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6296.561292] LustreError: Skipped 1 previous similar message [ 6296.562842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6296.592515] Lustre: Skipped 5 previous similar messages [ 6299.577546] Lustre: server umount lustre-OST0000 complete [ 6309.927518] Lustre: server umount lustre-MDT0000 complete [ 6311.917076] LustreError: 211833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6311.934709] LustreError: 211833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6313.799143] LustreError: 211814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385761 with bad export cookie 10495418443740290175 [ 6313.801730] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6313.810145] LustreError: 211814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6314.271719] Lustre: server umount lustre-MDT0001 complete [ 6322.936446] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 04:02:48 (1787385768) [ 6329.584669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6330.196536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6330.211555] Lustre: Skipped 6 previous similar messages [ 6334.847288] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6342.757186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.534904] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6355.446849] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6361.009708] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6368.076537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6375.645377] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6383.288037] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6388.287547] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6388.508500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6393.093150] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6393.355108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6399.880607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6400.048519] Lustre: lustre-OST0001: new disk, initializing [ 6400.058700] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6400.071344] Lustre: 221187:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6406.591685] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6407.772492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6407.792755] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6407.914549] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6416.256753] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6421.018343] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6421.226275] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6425.457523] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6425.661020] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6428.649534] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6428.658970] Lustre: Skipped 12 previous similar messages [ 6433.496769] Lustre: server umount lustre-OST0001 complete [ 6439.272683] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6439.606544] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6444.118543] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6444.310707] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6474.196205] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6479.862738] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6479.882857] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6479.886405] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6479.904154] Lustre: Skipped 1 previous similar message [ 6480.800545] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6488.171486] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6490.218953] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6495.205548] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6495.215992] LustreError: Skipped 1 previous similar message [ 6498.379388] Lustre: server umount lustre-OST0001 complete [ 6521.631396] Lustre: server umount lustre-OST0000 complete [ 6540.256157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6540.550893] Lustre: server umount lustre-MDT0000 complete [ 6544.926570] LustreError: 219660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385992 with bad export cookie 10495418443740291141 [ 6544.927388] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6544.944130] LustreError: 219660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6551.343468] Lustre: server umount lustre-MDT0001 complete [ 6576.778770] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:07:02 (1787386022) [ 6583.269836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6583.597427] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6583.634651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6587.728306] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6598.781693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6598.925286] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6598.979299] Lustre: 226433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6603.937776] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6611.718371] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6618.329742] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6625.930831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6626.304299] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6636.379101] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6644.872247] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6656.713191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6656.829411] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6664.471669] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6673.258996] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6675.049761] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6679.559849] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6679.758728] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6685.899907] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6686.229866] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6693.794300] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9aeb4627e000.ost_server_uuid 50 [ 6696.058877] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9aeb4627e000.ost_server_uuid in IDLE state after 0 sec [ 6696.966545] Lustre: Permanently deactivating lustre-OST0001 [ 6697.015484] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6702.228536] Lustre: Permanently reactivating lustre-OST0001 [ 6702.252162] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6702.260975] Lustre: Skipped 2 previous similar messages [ 6708.204649] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6708.241034] Lustre: Skipped 10 previous similar messages [ 6712.926812] Lustre: server umount lustre-OST0001 complete [ 6715.369631] LustreError: 228278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6715.394811] LustreError: 228278:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [ 6723.793598] Lustre: server umount lustre-OST0000 complete [ 6734.230622] Lustre: server umount lustre-MDT0000 complete [ 6738.376841] LustreError: 226424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386186 with bad export cookie 10495418443740292429 [ 6738.382189] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6738.388913] LustreError: 226424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6738.956658] Lustre: server umount lustre-MDT0001 complete [ 6767.566507] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:10:12 (1787386212) [ 6774.619762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6774.841408] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6774.866196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6774.877909] Lustre: Skipped 2 previous similar messages [ 6779.362708] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6786.989485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6787.083647] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6787.114243] Lustre: 233129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6791.226715] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6797.976868] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6803.104266] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6809.630634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6809.835604] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6809.853739] Lustre: Permanently deactivating lustre-OST0000 [ 6809.863496] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6816.298980] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6816.379428] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6825.340307] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6833.657675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6833.727364] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6841.100720] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6841.220513] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6848.995756] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6850.920582] Lustre: Permanently reactivating lustre-OST0000 [ 6850.944754] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6850.949726] Lustre: Skipped 2 previous similar messages [ 6859.827281] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6859.828214] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 43 seconds. I think it's dead, and I am evicting it. exp 0000000007d582ba, cur 1787386307 deadline 1787386294 last 1787386264 [ 6859.845702] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6859.861287] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6859.904186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6866.917548] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6866.923702] LustreError: Skipped 4 previous similar messages [ 6871.587622] Lustre: server umount lustre-OST0001 complete [ 6881.255708] Lustre: server umount lustre-OST0000 complete [ 6890.886456] Lustre: server umount lustre-MDT0000 complete [ 6894.349835] LustreError: 233121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386342 with bad export cookie 10495418443740294095 [ 6894.358687] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6894.361138] LustreError: 233121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6894.876489] Lustre: server umount lustre-MDT0001 complete [ 6904.893068] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 6907.751837] Key type lgssc unregistered [ 6908.068810] LNet: 238013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6908.076658] LNetError: 238013:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6908.094475] LNet: Removed LNI 192.168.203.156@tcp [ 6908.984477] Key type .llcrypt unregistered [ 6908.987650] Key type ._llcrypt unregistered [ 6934.088459] Key type ._llcrypt registered [ 6934.091456] Key type .llcrypt registered [ 6934.180877] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 6935.092063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6935.146443] alg: No test for adler32 (adler32-zlib) [ 6936.164185] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 6936.442722] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 6938.168168] Key type lgssc registered [ 6939.245211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6950.188161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6950.230664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6951.834511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6957.684549] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6962.150530] LustreError: 239479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6962.204708] LustreError: 239479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6966.379058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.532870] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6966.550164] Lustre: 239469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6966.557944] Lustre: Permanently deactivating lustre-MDT0001 [ 6966.569844] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6966.859188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6969.814384] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6969.824883] LustreError: 239510:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6969.836368] LustreError: 239510:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6969.847773] LustreError: 239510:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 6971.606971] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6980.372870] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6987.685161] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6995.358367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6995.634521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6996.655840] LustreError: 241381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6996.680184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7002.088416] LustreError: 241381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7002.495111] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7007.207414] LustreError: 241324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7010.066698] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7011.316431] LustreError: 241322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7011.346176] LustreError: 241322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7017.377899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7017.673262] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7023.105250] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7023.134246] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7024.251888] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7031.580868] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7034.737875] Lustre: Permanently reactivating lustre-MDT0001 [ 7034.773096] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7034.784415] Lustre: Skipped 2 previous similar messages [ 7041.408453] Lustre: lustre-MDT0001: Client 8066e404-375e-4cdc-b3db-32d26ddc8837 (at 192.168.203.56@tcp) reconnecting [ 7047.482972] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7049.169701] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7054.970226] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7055.187264] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7059.313872] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7059.494882] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7062.200181] Lustre: Permanently deactivating lustre-MDT0001 [ 7062.223795] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7062.229132] Lustre: Skipped 2 previous similar messages [ 7069.755307] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7079.394384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7079.399922] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7079.410196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7079.436983] Lustre: Skipped 3 previous similar messages [ 7084.514277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7084.519932] Lustre: Skipped 6 previous similar messages [ 7085.276322] Lustre: server umount lustre-MDT0000 complete [ 7088.673270] LustreError: 239461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386536 with bad export cookie 15572396944821853984 [ 7088.676720] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7088.686156] LustreError: 239461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7088.976182] Lustre: server umount lustre-MDT0001 complete [ 7094.175344] Lustre: server umount lustre-OST0000 complete [ 7104.603000] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:15:50 (1787386550) [ 7114.665614] Lustre: server umount lustre-OST0001 complete [ 7120.925667] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 7129.878814] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 7165.055576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7165.290255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7165.300400] Lustre: Skipped 2 previous similar messages [ 7165.333770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7165.442862] Lustre: lustre-MDT0000: new disk, initializing [ 7165.544776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7165.584679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7169.954737] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7180.231800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7180.364329] Lustre: 247759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7180.411535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7180.415825] Lustre: Skipped 1 previous similar message [ 7180.500575] Lustre: lustre-MDT0001: new disk, initializing [ 7180.603372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7180.660617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7180.684991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7184.774330] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7194.479183] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7200.962450] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7207.808465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7208.203453] Lustre: lustre-OST0000: new disk, initializing [ 7208.211295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7208.219030] Lustre: 249724:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7208.301638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.612546] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7216.190600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7216.205943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7216.313941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7223.752175] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7225.758445] LustreError: 247764:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7235.106651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7235.319198] Lustre: lustre-OST0001: new disk, initializing [ 7235.329581] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7235.339355] Lustre: 250839:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7235.784112] LustreError: 247764:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7235.804345] LustreError: 247766:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7240.835962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7240.850623] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7240.989139] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7242.853771] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7245.834346] LustreError: 247766:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7255.494209] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7257.337271] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7261.683855] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7261.684300] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7261.700049] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7261.752936] Lustre: Skipped 1 previous similar message [ 7265.695859] Lustre: server umount lustre-OST0001 complete [ 7266.787775] LustreError: 249709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7266.827471] LustreError: 249709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7283.170646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7283.180452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.182567] LustreError: Skipped 1 previous similar message [ 7283.184045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7283.201877] Lustre: Skipped 1 previous similar message [ 7283.218535] Lustre: Skipped 2 previous similar messages [ 7287.274527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7287.285329] Lustre: Skipped 1 previous similar message [ 7287.290120] LustreError: 249707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7287.315281] LustreError: 249707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 7289.691875] Lustre: server umount lustre-OST0000 complete [ 7297.510445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7297.534531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7297.543017] Lustre: Skipped 1 previous similar message [ 7300.700429] Lustre: server umount lustre-MDT0000 complete [ 7304.593104] LustreError: 249723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386752 with bad export cookie 15572396944821856581 [ 7304.604263] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7304.605053] LustreError: 249723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7311.045623] Lustre: server umount lustre-MDT0001 complete [ 7322.783816] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 7326.091452] Key type lgssc unregistered [ 7326.398712] LNet: 252873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7326.409781] LNetError: 252873:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7326.426259] LNet: Removed LNI 192.168.203.156@tcp [ 7327.685967] Key type .llcrypt unregistered [ 7327.692444] Key type ._llcrypt unregistered [ 7379.434882] Key type ._llcrypt registered [ 7379.437384] Key type .llcrypt registered [ 7379.576915] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 7382.262488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7382.358676] alg: No test for adler32 (adler32-zlib) [ 7383.600353] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 7383.955399] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 7385.664200] Key type lgssc registered [ 7387.473904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7396.644945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7396.660066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7397.922874] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7397.972073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7398.213816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7403.526449] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7414.346418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7414.496472] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7414.590357] Lustre: 255085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7415.063880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7422.469418] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7434.205921] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7443.797419] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7452.168730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7452.338557] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7452.577111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.325977] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7462.214729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7470.172269] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7528.931971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7528.940099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7528.956569] Lustre: Skipped 1 previous similar message [ 7528.962551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7528.996889] Lustre: Skipped 1 previous similar message [ 7533.464704] Lustre: server umount lustre-OST0000 complete [ 7538.581675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7553.873385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7554.503791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7562.829762] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7568.877741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7568.903883] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7568.916360] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7568.930156] Lustre: Skipped 1 previous similar message [ 7570.630387] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7614.948056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7614.951172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7614.962156] LustreError: Skipped 1 previous similar message [ 7614.969435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7614.975206] Lustre: Skipped 1 previous similar message [ 7616.227944] Lustre: server umount lustre-OST0000 complete [ 7620.067209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7620.067214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7620.068090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7620.118063] Lustre: Skipped 2 previous similar messages [ 7625.189473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7625.195879] Lustre: Skipped 1 previous similar message [ 7630.309761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7630.320195] Lustre: Skipped 1 previous similar message [ 7634.400147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7634.589049] Lustre: server umount lustre-MDT0000 complete [ 7638.694937] LustreError: 256951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387086 with bad export cookie 135777778987738286 [ 7638.700060] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7638.707668] LustreError: 256951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7639.168684] Lustre: server umount lustre-MDT0001 complete [ 7651.331338] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 7654.203306] Key type lgssc unregistered [ 7654.577631] LNet: 260515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7654.594162] LNetError: 260515:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7654.633796] LNet: Removed LNI 192.168.203.156@tcp [ 7656.155289] Key type .llcrypt unregistered [ 7656.158849] Key type ._llcrypt unregistered [ 7682.253872] Key type ._llcrypt registered [ 7682.256642] Key type .llcrypt registered [ 7682.401792] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 7683.573215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7683.607128] alg: No test for adler32 (adler32-zlib) [ 7684.800617] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 7685.191912] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 7686.944187] Key type lgssc registered [ 7688.616887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7697.741274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7697.761988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7699.205370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7703.932387] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7709.666563] LustreError: 261938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7709.688039] LustreError: 261938:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7714.122354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7714.621741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7720.087805] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7728.873186] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7734.823873] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7742.405510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7742.932211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7744.977944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7750.043038] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7758.559728] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7785.968107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7785.974655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7785.995482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7790.060358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7790.069622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7790.088395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7792.114153] Lustre: server umount lustre-OST0000 complete [ 7800.289700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.294530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7800.319450] Lustre: Skipped 1 previous similar message [ 7800.357081] Lustre: Skipped 2 previous similar messages [ 7803.667945] Lustre: server umount lustre-MDT0000 complete [ 7805.429165] LustreError: 261933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7805.462534] LustreError: 261933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7807.726387] LustreError: 261918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387255 with bad export cookie 4077258787305572885 [ 7807.738266] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7808.075224] Lustre: server umount lustre-MDT0001 complete [ 7820.168346] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 7823.228962] Key type lgssc unregistered [ 7823.584888] LNet: 266244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7823.595600] LNetError: 266244:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7823.623862] LNet: Removed LNI 192.168.203.156@tcp [ 7824.759464] Key type .llcrypt unregistered [ 7824.764118] Key type ._llcrypt unregistered [ 7845.386810] Key type ._llcrypt registered [ 7845.389676] Key type .llcrypt registered [ 7845.545605] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 7847.551301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7847.658472] alg: No test for adler32 (adler32-zlib) [ 7848.826467] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 7849.275145] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 7851.128309] Key type lgssc registered [ 7853.254167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7866.230453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7866.258118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7868.011162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7874.170630] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7877.184380] LustreError: 267466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7877.202839] LustreError: 267466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7882.210825] LustreError: 267467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7885.038938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7885.607120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7892.763370] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7901.897890] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7909.311455] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7917.347668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7917.572499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7925.207285] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7933.137710] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7938.047563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7951.334608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7951.346064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7951.364551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7952.036302] Lustre: server umount lustre-OST0000 complete [ 7957.475799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7957.484936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7957.503804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7958.521525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7958.542195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7958.550027] Lustre: Skipped 1 previous similar message [ 7963.617694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7963.629862] Lustre: Skipped 1 previous similar message [ 7966.787051] Lustre: server umount lustre-MDT0000 complete [ 7968.739316] LustreError: 267462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7968.758902] LustreError: 267462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7971.992284] LustreError: 267448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387419 with bad export cookie 13348620179404995 [ 7972.003423] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7972.012786] LustreError: 267448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7972.667735] Lustre: server umount lustre-MDT0001 complete [ 7986.665658] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 7990.224987] Key type lgssc unregistered [ 7990.580253] LNet: 271148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7990.587283] LNetError: 271148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7990.614259] LNet: Removed LNI 192.168.203.156@tcp [ 7991.518890] Key type .llcrypt unregistered [ 7991.527959] Key type ._llcrypt unregistered [ 8018.493450] Key type ._llcrypt registered [ 8018.495574] Key type .llcrypt registered [ 8018.624630] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 8020.107059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8020.163972] alg: No test for adler32 (adler32-zlib) [ 8021.324812] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 8021.658084] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 8023.320240] Key type lgssc registered [ 8024.741148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8033.107496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8033.134261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8034.557634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8039.838143] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8045.028275] LustreError: 272565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8045.056747] LustreError: 272565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8050.166216] LustreError: 272566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.433052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8050.949363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8055.910785] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8064.400427] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8070.319690] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8077.693816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8078.104053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8080.190950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8085.024240] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8093.377364] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8126.435469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8126.445957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8126.451937] LustreError: Skipped 1 previous similar message [ 8126.467292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8126.477645] Lustre: Skipped 1 previous similar message [ 8127.640234] Lustre: server umount lustre-OST0000 complete [ 8133.091486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8133.111544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8133.139290] Lustre: Skipped 1 previous similar message [ 8133.150290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8136.674617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8136.675797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8136.722678] Lustre: Skipped 1 previous similar message [ 8141.811733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8141.829866] Lustre: Skipped 1 previous similar message [ 8146.402654] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8146.693681] Lustre: server umount lustre-MDT0000 complete [ 8146.945307] LustreError: 272566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8146.970321] LustreError: 272566:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8151.085910] LustreError: 272546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387598 with bad export cookie 6781455623097274361 [ 8151.094905] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8151.097504] LustreError: 272546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8151.362888] Lustre: server umount lustre-MDT0001 complete [ 8163.981143] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 8167.014753] Key type lgssc unregistered [ 8167.356718] LNet: 276878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8167.375786] LNetError: 276878:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8167.399579] LNet: Removed LNI 192.168.203.156@tcp [ 8168.407103] Key type .llcrypt unregistered [ 8168.408659] Key type ._llcrypt unregistered [ 8191.718044] Key type ._llcrypt registered [ 8191.728645] Key type .llcrypt registered [ 8191.934712] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 8193.690637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8194.029215] alg: No test for adler32 (adler32-zlib) [ 8195.564521] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 8196.156074] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 8198.048597] Key type lgssc registered [ 8200.725549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8211.155519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8211.199471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8213.044240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8219.057805] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8222.192985] LustreError: 278125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8222.245036] LustreError: 278125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8227.298169] LustreError: 278127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8229.506478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8229.990400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8235.227398] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8243.102455] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8249.040639] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8255.673269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8256.275365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8257.334813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8263.365191] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8273.540240] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8290.275774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8290.279781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8290.294480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8290.296200] Lustre: Skipped 1 previous similar message [ 8294.762875] Lustre: server umount lustre-OST0000 complete [ 8300.516953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8300.532456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8300.537911] Lustre: Skipped 2 previous similar messages [ 8305.634157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8310.760791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8310.779670] Lustre: Skipped 2 previous similar messages [ 8313.312140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8313.619494] Lustre: server umount lustre-MDT0000 complete [ 8315.875339] LustreError: 278137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8315.893185] LustreError: 278137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8317.635325] LustreError: 278090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387765 with bad export cookie 180992988785816468 [ 8317.638944] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8317.643559] LustreError: 278090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8317.995265] Lustre: server umount lustre-MDT0001 complete [ 8329.446495] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 8331.972962] Key type lgssc unregistered [ 8332.226861] LNet: 281827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8332.231596] LNetError: 281827:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8332.252088] LNet: Removed LNI 192.168.203.156@tcp [ 8333.173847] Key type .llcrypt unregistered [ 8333.176089] Key type ._llcrypt unregistered [ 8346.079986] Key type ._llcrypt registered [ 8346.098356] Key type .llcrypt registered [ 8346.285195] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8366.272453] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 8381.489725] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 8383.265555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8383.338800] alg: No test for adler32 (adler32-zlib) [ 8384.687662] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 8385.090549] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 8386.976129] Key type lgssc registered [ 8388.996420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8430.066784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8430.082529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8431.287471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8431.314640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8431.380318] Lustre: lustre-MDT0000: new disk, initializing [ 8431.471251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8431.519173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8435.976896] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8446.250595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8446.472549] Lustre: 285387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8446.583097] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8446.595256] Lustre: Skipped 1 previous similar message [ 8446.746277] Lustre: lustre-MDT0001: new disk, initializing [ 8446.854305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8446.886716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8446.901423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8452.622506] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8464.293741] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8470.767714] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8478.739562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8479.022567] Lustre: lustre-OST0000: new disk, initializing [ 8479.026164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8479.037647] Lustre: 287354:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8479.153465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8485.546323] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8485.932576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8485.940802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8486.100047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8495.241735] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8500.488362] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8500.738335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8505.474987] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8505.697341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8511.970949] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8511.979866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8511.986384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8512.002322] Lustre: Skipped 1 previous similar message [ 8513.806657] Lustre: server umount lustre-OST0000 complete [ 8518.625440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8518.639541] LustreError: Skipped 1 previous similar message [ 8518.650402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8518.666407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8518.669421] Lustre: Skipped 1 previous similar message [ 8522.234647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8522.236082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8524.281648] Lustre: server umount lustre-MDT0000 complete [ 8528.253394] LustreError: 285377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387976 with bad export cookie 13082811413913237935 [ 8528.259401] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8528.267668] LustreError: 285377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8528.714589] Lustre: server umount lustre-MDT0001 complete [ 8538.221270] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 04:39:43 (1787387983) [ 8545.517750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8546.142377] LustreError: 289766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8546.164366] LustreError: 289766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8546.251038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8551.173557] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8551.397437] LustreError: 289767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8556.515123] LustreError: 289766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8560.576059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8561.111675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8566.791573] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8575.051404] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8581.676806] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8588.745283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8589.154393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8595.347291] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8603.412433] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8615.909922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8615.910514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8615.974364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8615.986990] Lustre: Skipped 2 previous similar messages [ 8620.971683] Lustre: server umount lustre-OST0000 complete [ 8626.165640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8626.174428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8626.206723] Lustre: Skipped 2 previous similar messages [ 8626.229873] Lustre: Skipped 1 previous similar message [ 8631.676601] Lustre: server umount lustre-MDT0000 complete [ 8635.261516] LustreError: 289748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388083 with bad export cookie 13082811413913238810 [ 8635.265968] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8635.279842] LustreError: 289748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8635.643069] Lustre: server umount lustre-MDT0001 complete [ 8645.649557] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 8648.890321] Key type lgssc unregistered [ 8649.291449] LNet: 293303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8649.311493] LNetError: 293303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8649.336811] LNet: Removed LNI 192.168.203.156@tcp [ 8650.143951] Key type .llcrypt unregistered [ 8650.148269] Key type ._llcrypt unregistered [ 8676.110374] Key type ._llcrypt registered [ 8676.113793] Key type .llcrypt registered [ 8676.227269] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 8690.257401] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 8691.212210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8691.245190] alg: No test for adler32 (adler32-zlib) [ 8692.296184] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 8692.469810] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 8694.112426] Key type lgssc registered [ 8695.527535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8737.553033] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8737.601369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8739.132869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8739.163724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8739.259859] Lustre: lustre-MDT0000: new disk, initializing [ 8739.363807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8739.408283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8744.826952] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8756.573246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8756.745083] Lustre: 297132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8756.876919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8756.887265] Lustre: Skipped 1 previous similar message [ 8757.030852] Lustre: lustre-MDT0001: new disk, initializing [ 8757.202947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8757.249606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8757.267774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8761.818619] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8771.985560] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8777.555975] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8783.960878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8784.177368] Lustre: lustre-OST2710: new disk, initializing [ 8784.185515] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8784.192332] Lustre: 299097:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8784.337340] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8790.476453] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8792.150201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8792.163610] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8792.229177] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8799.761636] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8807.146378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8807.338776] Lustre: lustre-OST03e8: new disk, initializing [ 8807.344550] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8807.351110] Lustre: 300165:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8807.463563] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8813.686649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8813.704091] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8813.872556] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8813.887399] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8821.010382] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8828.581911] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8828.783301] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8833.549931] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8833.771274] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8837.880132] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8838.067767] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8842.337760] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8842.540229] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8849.387084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8849.399926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8849.461367] Lustre: server umount lustre-MDT0000 complete [ 8849.901042] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8849.929994] Lustre: Skipped 2 previous similar messages [ 8853.003400] LustreError: 297123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388300 with bad export cookie 2362135611945060080 [ 8853.005803] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8853.017461] LustreError: 297123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8853.537886] Lustre: server umount lustre-MDT0001 complete [ 8868.013923] Lustre: server umount lustre-OST2710 complete [ 8870.886458] Lustre: 294928:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388302/real 1787388302] req@00000000d3aba03f x1874212306355072/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787388318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8870.937487] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8872.162873] Lustre: 294929:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388304/real 1787388304] req@00000000a8a57da6 x1874212306355328/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787388320 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8872.340945] Lustre: server umount lustre-OST03e8 complete [ 8889.754263] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 8899.170848] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 8937.201940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8937.633141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8937.706311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8937.877614] Lustre: lustre-MDT0000: new disk, initializing [ 8938.028032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8938.070637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8943.275490] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8954.584739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8954.754870] Lustre: 305414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8954.816127] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8954.820233] Lustre: Skipped 1 previous similar message [ 8954.877786] Lustre: lustre-MDT0001: new disk, initializing [ 8955.015749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8955.088817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8955.108809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8959.892621] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8968.964244] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8974.241209] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8980.867606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8981.109256] Lustre: lustre-OST0000: new disk, initializing [ 8981.112717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8981.118067] Lustre: 307385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8981.203909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8987.127404] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8987.226212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8987.235571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8987.395077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8996.297053] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9000.822757] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9001.043561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9005.365475] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9005.579114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9008.098657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9008.100824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9008.106828] LustreError: Skipped 1 previous similar message [ 9008.107519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9008.142815] Lustre: Skipped 1 previous similar message [ 9013.224721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9013.241380] Lustre: Skipped 2 previous similar messages [ 9013.527897] Lustre: server umount lustre-OST0000 complete [ 9018.344711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9018.360789] Lustre: Skipped 1 previous similar message [ 9018.361114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9018.375393] Lustre: Skipped 1 previous similar message [ 9023.279972] Lustre: server umount lustre-MDT0000 complete [ 9023.463090] LustreError: 306225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9023.493397] LustreError: 306225:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9027.139517] LustreError: 305407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388474 with bad export cookie 2362135611945061760 [ 9027.158782] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9027.458030] Lustre: server umount lustre-MDT0001 complete [ 9036.907577] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:48:02 (1787388482) [ 9038.498240] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9040.130591] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 04:48:05 (1787388485) [ 9044.263683] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 9072.733611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9073.018186] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9073.045572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9073.265336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9078.061583] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9087.401939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9087.525464] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9087.554841] Lustre: 311242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9093.079167] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9116.545756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9116.906725] Lustre: Denying initial registration attempt from nid 192.168.203.156@tcp, specified as failover [ 9116.939947] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9116.951388] LustreError: 312696:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9116.957503] LustreError: 312696:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9116.966048] LustreError: 312696:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9117.154883] Lustre: server umount lustre-OST0000 complete [ 9117.167814] LustreError: 312696:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9125.348106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9125.355728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9125.368447] Lustre: Skipped 1 previous similar message [ 9125.385455] Lustre: Skipped 1 previous similar message [ 9130.469141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9130.479260] Lustre: Skipped 1 previous similar message [ 9131.632266] Lustre: server umount lustre-MDT0000 complete [ 9135.591471] LustreError: 312318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9135.600662] LustreError: 312318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9135.882355] LustreError: 311235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388583 with bad export cookie 2362135611945062642 [ 9135.890274] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9135.896635] LustreError: 311235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9136.312696] Lustre: server umount lustre-MDT0001 complete [ 9146.507657] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 9149.250871] Key type lgssc unregistered [ 9149.589789] LNet: 313818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9149.606657] LNetError: 313818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9149.628023] LNet: Removed LNI 192.168.203.156@tcp [ 9150.454100] Key type .llcrypt unregistered [ 9150.458581] Key type ._llcrypt unregistered [ 9161.304534] Key type ._llcrypt registered [ 9161.307851] Key type .llcrypt registered [ 9161.442450] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 9162.434085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9162.580947] alg: No test for adler32 (adler32-zlib) [ 9163.555931] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 9163.779613] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 9165.448220] Key type lgssc registered [ 9166.662123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9212.563753] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 9219.909198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9219.947777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9221.277534] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9221.316206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9221.485085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9225.318354] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9233.371606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9233.501783] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9233.540244] Lustre: 316589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9233.860685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9238.137067] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9244.588134] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9250.532834] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9269.639087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9269.870277] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9270.047895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9276.497177] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9283.887834] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9289.196695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9289.205466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9289.221275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9289.702596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9289.712560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9289.723993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9292.084339] Lustre: server umount lustre-OST0000 complete [ 9299.939775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9299.943348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9299.963828] Lustre: Skipped 1 previous similar message [ 9302.008073] Lustre: server umount lustre-MDT0000 complete [ 9305.664806] LustreError: 317329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388753 with bad export cookie 10046168052394119626 [ 9305.667698] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9305.676251] LustreError: 317329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9306.062810] Lustre: server umount lustre-MDT0001 complete [ 9315.306964] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 9317.808309] Key type lgssc unregistered [ 9318.083224] LNet: 320162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9318.092694] LNetError: 320162:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9318.109377] LNet: Removed LNI 192.168.203.156@tcp [ 9318.809982] Key type .llcrypt unregistered [ 9318.812386] Key type ._llcrypt unregistered [ 9339.859133] Key type ._llcrypt registered [ 9339.861947] Key type .llcrypt registered [ 9339.950862] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 9340.934120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9340.994991] alg: No test for adler32 (adler32-zlib) [ 9342.008475] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 9342.224096] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 9343.873306] Key type lgssc registered [ 9344.982798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9351.707460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9351.720487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9353.133079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9357.482570] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9362.405500] LustreError: 321559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9362.421635] LustreError: 321559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9365.454324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.807607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9369.860161] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9376.926821] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9382.190681] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9391.148793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9391.628677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9397.915194] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9404.373969] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9417.188066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9417.195346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9417.205762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9417.314217] Lustre: server umount lustre-MDT0000 complete [ 9420.367911] LustreError: 321539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388868 with bad export cookie 15950331222869003229 [ 9420.380487] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9420.381169] LustreError: 321539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9421.284994] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9421.305986] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9421.311054] LustreError: 324049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9425.313882] LustreError: 321554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9425.314631] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9425.328568] LustreError: 321554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9425.340052] Lustre: Skipped 1 previous similar message [ 9426.755220] Lustre: server umount lustre-MDT0001 complete [ 9432.236921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9439.175222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9445.858715] LustreError: 320868:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c50c6b15 x1874212987894400/t0(0) o250->MGC192.168.203.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9446.162724] LustreError: 323391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9446.233962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9446.282146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9450.348266] LustreError: 325170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9450.357757] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9450.640102] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9459.784357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9460.190169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9464.153617] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9465.340270] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9471.138588] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9472.665789] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9478.038053] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9479.693839] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9485.805914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9485.809738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9485.810314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9485.829804] Lustre: Skipped 2 previous similar messages [ 9490.613607] Lustre: server umount lustre-OST0000 complete [ 9496.033449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9496.066279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9496.071580] Lustre: Skipped 2 previous similar messages [ 9505.252210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9505.270976] Lustre: Skipped 3 previous similar messages [ 9508.321649] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9508.577931] Lustre: server umount lustre-MDT0000 complete [ 9510.374376] LustreError: 326628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9510.400544] LustreError: 326628:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9511.881398] LustreError: 321541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388959 with bad export cookie 15950331222869010369 [ 9511.883502] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9511.892901] LustreError: 321541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9512.152970] Lustre: server umount lustre-MDT0001 complete [ 9523.048963] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 9525.582925] Key type lgssc unregistered [ 9525.894289] LNet: 327844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9525.904848] LNetError: 327844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9525.935539] LNet: Removed LNI 192.168.203.156@tcp [ 9526.775713] Key type .llcrypt unregistered [ 9526.778944] Key type ._llcrypt unregistered [ 9554.350498] Key type ._llcrypt registered [ 9554.354979] Key type .llcrypt registered [ 9554.536043] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 9555.739372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9555.879952] alg: No test for adler32 (adler32-zlib) [ 9556.918537] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 9557.152472] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 9558.817699] Key type lgssc registered [ 9560.414481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9567.761600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9567.789441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9569.317416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9573.486414] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9579.489391] LustreError: 329262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9579.507699] LustreError: 329262:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9580.992489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9581.297135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9585.475310] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9591.776269] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9596.227550] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9599.972987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9599.988884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9599.993019] Lustre: Skipped 1 previous similar message [ 9600.007607] Lustre: Skipped 1 previous similar message [ 9605.093332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9605.104523] Lustre: Skipped 1 previous similar message [ 9607.140470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9607.146612] Lustre: Skipped 1 previous similar message [ 9612.260777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9612.268260] Lustre: Skipped 1 previous similar message [ 9613.280304] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9615.499137] Lustre: server umount lustre-MDT0000 complete [ 9617.379781] LustreError: 329261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9617.397601] LustreError: 329261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9618.705866] LustreError: 329996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389066 with bad export cookie 283652379639020115 [ 9618.711913] LustreError: 329996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9618.712737] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9619.075952] Lustre: server umount lustre-MDT0001 complete [ 9632.802405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9633.170992] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9633.214943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9633.427231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9638.046272] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9645.831566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9645.923543] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9645.965354] Lustre: 331848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9646.223848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9649.658846] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9655.367905] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9659.729901] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9668.843922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9669.084571] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9669.104706] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9669.121347] LustreError: 333732:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9669.131462] LustreError: 333732:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9669.139296] LustreError: 333732:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9669.206834] Lustre: server umount lustre-OST0000 complete [ 9669.213056] LustreError: 333732:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9678.967861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9679.286635] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9679.522277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9685.146097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9685.365407] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9692.394637] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9700.306206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9700.438443] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9700.554466] Lustre: lustre-OST0001: new disk, initializing [ 9700.560106] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9700.569546] Lustre: 335272:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9700.712427] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9706.606084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9706.622511] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9706.771041] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9707.956539] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9714.770309] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9722.338628] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9722.345473] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9722.351987] LustreError: Skipped 1 previous similar message [ 9722.364193] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9722.368123] Lustre: Skipped 1 previous similar message [ 9722.390677] Lustre: Skipped 1 previous similar message [ 9723.858996] Lustre: server umount lustre-OST0001 complete [ 9727.473541] LustreError: 334199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9727.494637] LustreError: 334199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9731.044861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9731.056393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9731.078260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9732.584637] LustreError: 335563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9732.613042] LustreError: 335563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9733.886717] Lustre: server umount lustre-OST0000 complete [ 9737.701216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9737.723086] Lustre: Skipped 2 previous similar messages [ 9743.890250] Lustre: server umount lustre-MDT0000 complete [ 9747.579656] LustreError: 331842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389195 with bad export cookie 283652379639020668 [ 9747.584699] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9747.596205] LustreError: 331842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9747.930602] Lustre: server umount lustre-MDT0001 complete [ 9758.352605] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 9761.310623] Key type lgssc unregistered [ 9761.650627] LNet: 337230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9761.658271] LNetError: 337230:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9761.675592] LNet: Removed LNI 192.168.203.156@tcp [ 9762.538355] Key type .llcrypt unregistered [ 9762.541557] Key type ._llcrypt unregistered [ 9821.982467] Key type ._llcrypt registered [ 9821.988163] Key type .llcrypt registered [ 9822.090408] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 9834.588763] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 9835.670412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9835.706214] alg: No test for adler32 (adler32-zlib) [ 9836.737859] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 9837.025108] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 9838.800207] Key type lgssc registered [ 9839.886322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9868.941908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9868.968101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9870.209767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9870.245791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9870.331801] Lustre: lustre-MDT0000: new disk, initializing [ 9870.396807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9870.412949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9874.266682] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9882.722942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9882.822116] Lustre: 342153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9882.876661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9882.881406] Lustre: Skipped 1 previous similar message [ 9882.942779] Lustre: lustre-MDT0001: new disk, initializing [ 9883.024583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9883.052070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9883.059603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9886.555295] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9894.561609] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9899.602356] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9906.792457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9907.084384] Lustre: lustre-OST0000: new disk, initializing [ 9907.089554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9907.099155] Lustre: 344121:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9907.187281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9912.690873] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9916.462136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9916.478574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9916.608598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9920.790716] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9924.852721] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9925.080080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9928.806678] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9929.053455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9932.260539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9932.269714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9932.283731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9932.286609] Lustre: Skipped 1 previous similar message [ 9936.821262] Lustre: server umount lustre-OST0000 complete [ 9942.498712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9942.500731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9942.508703] Lustre: Skipped 2 previous similar messages [ 9946.981928] Lustre: server umount lustre-MDT0000 complete [ 9950.727450] LustreError: 342146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389398 with bad export cookie 10298833350531421114 [ 9950.740431] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9950.749347] LustreError: 342146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9951.098680] Lustre: server umount lustre-MDT0001 complete [ 9958.063368] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:03:23 (1787389403) [ 9964.461949] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:03:30 (1787389410) [ 9972.234988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9972.712811] LustreError: 346875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9972.735275] LustreError: 346875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9972.822546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9976.765548] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9977.825875] LustreError: 346876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9982.949707] LustreError: 346875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9985.123940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9985.593382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9989.802662] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9996.474991] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10001.888388] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10008.286933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10008.605338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10013.952279] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10019.796185] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10027.442870] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10028.956154] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10030.287759] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10031.874769] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10033.863430] Lustre: Failing over lustre-MDT0000 [10034.356363] Lustre: server umount lustre-MDT0000 complete [10036.705855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10036.718346] LustreError: Skipped 1 previous similar message [10036.729797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.748529] LustreError: 346875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10042.702889] LustreError: 346871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10042.727426] LustreError: 346871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10052.683336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10052.773506] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10053.009177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10053.056633] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10056.988631] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10058.057248] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10058.229262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10058.256117] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10058.302197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10066.660169] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10068.367808] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10071.629699] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10073.492109] Lustre: Failing over lustre-MDT0000 [10073.569058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10073.572924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10073.580886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10073.580894] Lustre: Skipped 1 previous similar message [10073.614710] Lustre: Skipped 3 previous similar messages [10073.975902] Lustre: server umount lustre-MDT0000 complete [10078.579221] LustreError: 346872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10078.594072] LustreError: 346872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10094.775255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10094.969672] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10095.041884] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.56@tcp (not set up) [10095.049789] Lustre: Skipped 2 previous similar messages [10095.301710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10095.360873] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10100.098234] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10100.557640] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10100.732700] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10100.738082] Lustre: Skipped 2 previous similar messages [10100.772242] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10100.808185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10109.365444] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10111.106081] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10121.184949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10121.191132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10121.193967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10121.214345] Lustre: Skipped 3 previous similar messages [10123.250387] Lustre: server umount lustre-MDT0000 complete [10126.305708] LustreError: 346872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10126.326355] LustreError: 346872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [10126.987763] LustreError: 346856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389574 with bad export cookie 10298833350531423823 [10127.011280] LustreError: 346856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10127.014969] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10127.459480] Lustre: server umount lustre-MDT0001 complete [10141.811198] Lustre: server umount lustre-OST0000 complete [10157.812503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10162.459935] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10163.686270] LustreError: 353213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10163.704498] LustreError: 353213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10170.759836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10171.265982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10171.270911] Lustre: Skipped 1 previous similar message [10175.781408] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10182.981809] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10189.131859] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10195.467287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10201.627171] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10204.155054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10208.652453] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10214.369847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10214.387270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10214.393567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10214.416315] Lustre: Skipped 1 previous similar message [10214.445854] Lustre: Skipped 3 previous similar messages [10219.250808] Lustre: server umount lustre-OST0000 complete [10229.416069] Lustre: server umount lustre-MDT0000 complete [10229.730921] LustreError: 354638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10229.756315] LustreError: 354638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10232.627079] LustreError: 355793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389680 with bad export cookie 10298833350531424572 [10232.629130] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10232.640854] LustreError: 355793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10232.947481] Lustre: server umount lustre-MDT0001 complete [10244.378802] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [10247.193579] Key type lgssc unregistered [10247.489235] LNet: 356747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10247.500398] LNetError: 356747:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10248.554454] LNet: Removed LNI 192.168.203.156@tcp [10249.559603] Key type .llcrypt unregistered [10249.562262] Key type ._llcrypt unregistered [10278.562851] Key type ._llcrypt registered [10278.569822] Key type .llcrypt registered [10278.721560] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [10279.782928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10279.876535] alg: No test for adler32 (adler32-zlib) [10280.932499] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [10281.151411] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [10282.801597] Key type lgssc registered [10284.140921] Lustre: Echo OBD driver; http://www.lustre.org/ [10290.909071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10290.925250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10292.315487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10295.858987] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10302.438696] LustreError: 358265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10302.453757] LustreError: 358265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10302.544812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10302.789156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10306.328993] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10312.969837] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10317.384532] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10323.800357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10324.088773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10329.449206] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10335.990121] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10337.394249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10348.008577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10348.009944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10348.020561] Lustre: Skipped 2 previous similar messages [10348.027093] Lustre: Skipped 2 previous similar messages [10351.393792] Lustre: server umount lustre-MDT0000 complete [10353.129984] LustreError: 358264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10353.146895] LustreError: 358264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10354.301344] LustreError: 358997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389802 with bad export cookie 12519568674502052038 [10354.303873] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10354.318196] LustreError: 358997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10354.726832] Lustre: server umount lustre-MDT0001 complete [10367.452948] Lustre: server umount lustre-OST0000 complete [10377.671660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10378.011399] LustreError: 361972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10378.069729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10381.831657] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10383.331391] LustreError: 361973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10387.499956] LustreError: 361972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10389.849149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10390.370522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10395.159533] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10403.293431] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10408.753235] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10414.478558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10414.788385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10420.076418] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10423.020728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10425.786992] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10438.625364] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10438.631466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10438.640338] Lustre: Skipped 2 previous similar messages [10438.656029] Lustre: Skipped 2 previous similar messages [10443.117078] Lustre: server umount lustre-MDT0000 complete [10443.752943] LustreError: 361972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10443.770742] LustreError: 361972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10446.302248] LustreError: 364566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389894 with bad export cookie 12519568674502053151 [10446.304287] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10446.317447] LustreError: 364566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10446.665120] Lustre: server umount lustre-MDT0001 complete [10460.554471] Lustre: server umount lustre-OST0000 complete [10473.194641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10473.539278] LustreError: 365882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10473.616560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10477.556824] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10486.014146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10491.381909] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10499.661056] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10506.683133] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10514.301711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10514.949896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10514.960496] Lustre: Skipped 1 previous similar message [10519.039835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10521.533978] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10528.669834] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10534.370123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10534.383920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10534.407717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10536.423531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10536.426111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10536.457988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10538.920666] Lustre: server umount lustre-OST0000 complete [10546.658453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10546.668476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10546.682232] Lustre: Skipped 1 previous similar message [10546.704532] Lustre: Skipped 2 previous similar messages [10549.235979] Lustre: server umount lustre-MDT0000 complete [10551.777904] LustreError: 365878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10551.790723] LustreError: 365878:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10552.955121] LustreError: 365864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390000 with bad export cookie 12519568674502054173 [10552.983164] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10553.356870] Lustre: server umount lustre-MDT0001 complete [10563.284893] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [10565.765753] Key type lgssc unregistered [10566.016057] LNet: 369414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10566.032993] LNetError: 369414:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10566.063684] LNet: Removed LNI 192.168.203.156@tcp [10566.744362] Key type .llcrypt unregistered [10566.747207] Key type ._llcrypt unregistered [10590.469688] Key type ._llcrypt registered [10590.475447] Key type .llcrypt registered [10590.612276] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [10591.626487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10591.642828] alg: No test for adler32 (adler32-zlib) [10592.658072] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [10592.846297] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [10594.514197] Key type lgssc registered [10595.783124] Lustre: Echo OBD driver; http://www.lustre.org/ [10603.491160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10603.512494] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10603.521963] LustreError: 370854:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10603.548027] LustreError: 370854:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10603.561804] LustreError: 370854:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10603.572952] LustreError: 370854:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10603.579314] LustreError: 370854:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10610.105891] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10610.111345] LustreError: 371251:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10610.134065] LustreError: 371251:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10610.141254] LustreError: 371251:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10610.146220] LustreError: 371251:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10610.153058] LustreError: 371251:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10627.227187] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [10629.578911] Key type lgssc unregistered [10629.811700] LNet: 372253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10629.815428] LNetError: 372253:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10629.831714] LNet: Removed LNI 192.168.203.156@tcp [10630.655469] Key type .llcrypt unregistered [10630.657575] Key type ._llcrypt unregistered [10645.275779] Key type ._llcrypt registered [10645.278168] Key type .llcrypt registered [10645.368301] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [10656.729505] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [10657.591975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10657.750469] alg: No test for adler32 (adler32-zlib) [10658.757776] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [10659.013930] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [10660.696177] Key type lgssc registered [10661.927848] Lustre: Echo OBD driver; http://www.lustre.org/ [10692.032665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10692.050857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10693.335159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10693.363022] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10693.462265] Lustre: lustre-MDT0000: new disk, initializing [10693.580525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10693.595835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10697.783824] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10707.094794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10707.206543] Lustre: 375388:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10707.249418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10707.253342] Lustre: Skipped 1 previous similar message [10707.307930] Lustre: lustre-MDT0001: new disk, initializing [10707.385423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10707.413209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10707.423830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10711.370758] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10720.060784] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10725.275311] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10730.799614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10731.064172] Lustre: lustre-OST0000: new disk, initializing [10731.069925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10731.081047] Lustre: 377359:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10731.158958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10736.720517] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10738.749317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10738.761496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10738.842148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10745.403295] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10749.847380] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10750.059720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10754.042926] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10754.225983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10759.655728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10759.665633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10759.666983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10759.667072] Lustre: Skipped 1 previous similar message [10759.690912] Lustre: Skipped 1 previous similar message [10764.772850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10764.785318] Lustre: Skipped 1 previous similar message [10769.892952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10769.896308] Lustre: Skipped 1 previous similar message [10770.401707] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10770.644756] Lustre: server umount lustre-OST0000 complete [10773.987034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10773.995051] LustreError: Skipped 1 previous similar message [10773.998606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10774.013328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10775.012870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10780.132240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10780.139822] Lustre: Skipped 3 previous similar messages [10788.320670] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10788.515792] Lustre: server umount lustre-MDT0000 complete [10792.063305] LustreError: 375382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390239 with bad export cookie 14214571529008800471 [10792.075103] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10792.083247] LustreError: 375382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10792.537143] Lustre: server umount lustre-MDT0001 complete [10799.922105] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10801.338364] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:17:27 (1787390247) [10807.011719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10807.378356] LustreError: 379821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10807.399636] LustreError: 379821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10807.453830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10810.969760] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10812.900808] LustreError: 379822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10817.816132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.111399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10821.922642] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10829.634907] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10835.277491] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10841.431439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10841.787384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10848.178402] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10855.487586] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10861.016839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10861.154978] Lustre: lustre-OST0001: new disk, initializing [10861.160515] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10861.171568] Lustre: 382655:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10861.252599] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10866.397856] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10870.371872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10870.383483] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10870.471787] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10875.106344] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10880.993583] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10880.997454] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10881.002664] LustreError: Skipped 1 previous similar message [10881.003871] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10881.015360] Lustre: Skipped 1 previous similar message [10881.015798] Lustre: Skipped 2 previous similar messages [10885.792504] Lustre: server umount lustre-OST0001 complete [10886.126264] LustreError: 381641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10886.144176] LustreError: 381641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10891.241416] LustreError: 383184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10891.260167] LustreError: 383184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10896.358163] LustreError: 381641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10896.392976] LustreError: 381641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10901.478251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10901.499181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10901.512153] Lustre: Skipped 3 previous similar messages [10906.600516] LustreError: 383184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10906.634145] LustreError: 383184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10907.554595] Lustre: server umount lustre-OST0000 complete [10911.714143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10911.736114] Lustre: Skipped 2 previous similar messages [10916.765944] Lustre: server umount lustre-MDT0000 complete [10920.375430] LustreError: 379802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390368 with bad export cookie 14214571529008801353 [10920.378757] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10920.386191] LustreError: 379802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10926.913267] Lustre: server umount lustre-MDT0001 complete [10938.436176] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [10940.927282] Key type lgssc unregistered [10941.149636] LNet: 384642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10941.155291] LNetError: 384642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10941.181938] LNet: Removed LNI 192.168.203.156@tcp [10941.981313] Key type .llcrypt unregistered [10941.993235] Key type ._llcrypt unregistered [10984.746708] Key type ._llcrypt registered [10984.752392] Key type .llcrypt registered [10984.839976] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [10985.951259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10985.966083] alg: No test for adler32 (adler32-zlib) [10986.990658] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [10987.211841] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [10988.864231] Key type lgssc registered [10989.833679] Lustre: Echo OBD driver; http://www.lustre.org/ [10995.919365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10995.935446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10997.185553] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10997.210124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10997.334644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11000.617050] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11007.324934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11007.420608] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11007.447301] Lustre: 387418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11007.696275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11011.038047] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11017.045918] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11022.234204] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11026.406221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11026.423417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11026.426042] Lustre: Skipped 1 previous similar message [11026.447195] Lustre: Skipped 1 previous similar message [11030.343503] Lustre: server umount lustre-MDT0000 complete [11033.937087] LustreError: 387413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390481 with bad export cookie 2159165285649116755 [11033.940037] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11033.954951] LustreError: 387413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11034.268600] Lustre: server umount lustre-MDT0001 complete [11043.193453] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:21:28 (1787390488) [11049.203743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11049.719196] LustreError: 389932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11049.743243] LustreError: 389932:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11049.792761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11053.719163] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11055.075312] LustreError: 389933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11060.193199] LustreError: 389932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11061.918970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11062.182947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11066.658503] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11073.922830] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11080.128628] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11088.207882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11088.493421] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11088.823501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11096.995450] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11106.088496] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11114.525339] Lustre: Permanently deactivating lustre-OST0000 [11114.554574] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11122.216425] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11123.420753] LustreError: 392918:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11123.433557] LustreError: 392918:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11123.441019] LustreError: 392918:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11126.745751] LustreError: 392967:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11126.753776] LustreError: 392967:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11131.261567] Lustre: server umount lustre-OST0000 complete [11133.730820] LustreError: 393216:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11133.741315] LustreError: 393216:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11139.045947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11139.055163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11139.074055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11144.676942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11144.685132] Lustre: Skipped 2 previous similar messages [11145.759993] Lustre: server umount lustre-MDT0000 complete [11148.966355] LustreError: 389914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390596 with bad export cookie 2159165285649117336 [11148.969390] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11148.975316] LustreError: 389914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11158.694867] Lustre: server umount lustre-MDT0001 complete [11166.457495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11170.879613] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11173.481389] LustreError: 394375:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11179.572635] Lustre: 394714:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.156@tcp [11181.422970] Lustre: server umount MGS complete [11191.710768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11192.177324] LustreError: 395439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11192.196199] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11192.204628] Lustre: Skipped 1 previous similar message [11192.250981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11195.978582] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11197.420066] LustreError: 395439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11203.709555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11204.015933] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11208.679443] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11216.358813] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11222.076993] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11223.863873] Lustre: Permanently reactivating lustre-OST0000 [11223.901610] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11223.910827] Lustre: Skipped 2 previous similar messages [11253.216135] LustreError: 397267:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11260.416238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11260.693531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11260.702133] Lustre: Skipped 1 previous similar message [11266.066188] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11272.744410] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11278.308044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11278.315416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11278.336428] Lustre: Skipped 1 previous similar message [11278.345153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11279.343316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11283.900474] Lustre: server umount lustre-OST0000 complete [11289.578596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11289.585380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11289.594725] Lustre: Skipped 2 previous similar messages [11289.614343] Lustre: Skipped 3 previous similar messages [11293.372413] Lustre: server umount lustre-MDT0000 complete [11294.691910] LustreError: 395434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11294.703448] LustreError: 395434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11296.287413] LustreError: 395419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390744 with bad export cookie 2159165285649118386 [11296.291352] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11296.300300] LustreError: 395419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11296.545641] Lustre: server umount lustre-MDT0001 complete [11304.423074] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [11306.589332] Key type lgssc unregistered [11306.922561] LNet: 399897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11306.936164] LNetError: 399897:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11306.953109] LNet: Removed LNI 192.168.203.156@tcp [11307.636467] Key type .llcrypt unregistered [11307.639886] Key type ._llcrypt unregistered [11329.956661] Key type ._llcrypt registered [11329.960924] Key type .llcrypt registered [11330.103746] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [11348.847387] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [11349.965359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11349.978183] alg: No test for adler32 (adler32-zlib) [11350.984670] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [11351.212716] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [11352.864129] Key type lgssc registered [11354.015500] Lustre: Echo OBD driver; http://www.lustre.org/ [11389.106927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11389.153767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11390.396981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11390.456812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11390.539960] Lustre: lustre-MDT0000: new disk, initializing [11390.586382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11390.603609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11395.426789] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11406.469635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11406.681733] Lustre: 403057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11406.757637] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11406.770404] Lustre: Skipped 1 previous similar message [11406.859038] Lustre: lustre-MDT0001: new disk, initializing [11406.981802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11407.058394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11407.069278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11411.968029] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11421.809659] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11426.959906] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11433.239731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11433.422646] Lustre: lustre-OST0000: new disk, initializing [11433.428435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11433.437080] Lustre: 405024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11433.521128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11438.648333] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11439.200987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11439.221312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11439.308305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11448.259533] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11453.475365] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11453.717434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11457.847286] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11458.047492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11460.067654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11460.074250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11460.091683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11460.113369] LustreError: Skipped 1 previous similar message [11465.188914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11465.192439] Lustre: Skipped 1 previous similar message [11470.316412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11470.324744] Lustre: Skipped 1 previous similar message [11473.889468] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11474.021583] Lustre: server umount lustre-OST0000 complete [11478.499095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11478.509661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11478.539603] Lustre: Skipped 1 previous similar message [11478.569633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11480.545672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11485.672269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11485.681732] Lustre: Skipped 3 previous similar messages [11492.322080] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11492.531669] Lustre: server umount lustre-MDT0000 complete [11496.320118] LustreError: 403048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390944 with bad export cookie 11583881509840337623 [11496.324219] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11496.341778] LustreError: 403048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11496.709847] Lustre: server umount lustre-MDT0001 complete [11505.101210] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11506.826825] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:29:12 (1787390952) [11513.113062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11513.735559] LustreError: 407487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11513.754535] LustreError: 407487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11513.803496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11518.320739] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11518.946750] LustreError: 407488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11524.069781] LustreError: 407487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11526.128938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11526.556409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11531.280714] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11538.017957] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11543.564364] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11550.665429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11550.970316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11557.253493] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11564.892697] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11567.009152] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11569.538297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11574.766142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11574.769516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11574.777113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11574.777120] Lustre: Skipped 2 previous similar messages [11574.810507] Lustre: Skipped 1 previous similar message [11580.841510] Lustre: server umount lustre-OST0000 complete [11584.994485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11585.001548] Lustre: Skipped 1 previous similar message [11590.651806] Lustre: server umount lustre-MDT0000 complete [11594.293552] LustreError: 407468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391042 with bad export cookie 11583881509840338505 [11594.298281] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11594.304478] LustreError: 407468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11594.922474] Lustre: server umount lustre-MDT0001 complete [11606.268221] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [11609.367056] Key type lgssc unregistered [11609.669505] LNet: 411212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11609.679709] LNetError: 411212:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11609.696397] LNet: Removed LNI 192.168.203.156@tcp [11610.551653] Key type .llcrypt unregistered [11610.555019] Key type ._llcrypt unregistered [11636.921823] Key type ._llcrypt registered [11636.932956] Key type .llcrypt registered [11637.168357] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [11637.722898] Key type .llcrypt unregistered [11637.726559] Key type ._llcrypt unregistered [11660.893130] Key type ._llcrypt registered [11660.895885] Key type .llcrypt registered [11661.064153] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [11662.095140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11662.181775] alg: No test for adler32 (adler32-zlib) [11663.342868] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [11663.631796] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [11665.298712] Key type lgssc registered [11666.987956] Lustre: Echo OBD driver; http://www.lustre.org/ [11674.492829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11674.530522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11676.117388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11680.391671] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11685.346512] LustreError: 413427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11685.376087] LustreError: 413427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11687.742266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11688.000221] LustreError: 413423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11688.019012] LustreError: 413423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11688.019162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11693.031279] LustreError: 413428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11693.050308] LustreError: 413428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11693.827343] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11698.154526] LustreError: 413427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11698.173714] LustreError: 413427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11700.216988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11700.569725] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11705.319821] LustreError: 413428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11705.347630] LustreError: 413428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11706.972060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11707.420454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11710.835365] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11720.674228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11720.679150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11720.686150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11722.721557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11722.726046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11722.730511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11722.741531] Lustre: Skipped 1 previous similar message [11726.124893] Lustre: server umount lustre-OST0000 complete [11732.392870] Lustre: server umount lustre-MDT0000 complete [11732.960930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11732.962536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11732.987447] LustreError: 413427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11733.023034] Lustre: Skipped 1 previous similar message [11736.360872] LustreError: 415534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391184 with bad export cookie 1990163403576253301 [11736.362589] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11736.370438] LustreError: 415534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11736.888331] Lustre: server umount lustre-MDT0001 complete [11747.297457] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [11749.781556] Key type lgssc unregistered [11750.033493] LNet: 416573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11750.044574] LNetError: 416573:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11750.061612] LNet: Removed LNI 192.168.203.156@tcp [11750.839402] Key type .llcrypt unregistered [11750.841254] Key type ._llcrypt unregistered [11773.574234] Key type ._llcrypt registered [11773.576286] Key type .llcrypt registered [11773.699598] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [11774.802949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11774.898203] alg: No test for adler32 (adler32-zlib) [11775.932385] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [11776.204019] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [11777.856197] Key type lgssc registered [11778.910033] Lustre: Echo OBD driver; http://www.lustre.org/ [11786.811468] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11786.831092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11802.336570] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11807.456508] LustreError: 417968:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.156@tcp: failed processing log, type 4: rc = -110 [11834.208413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11839.514339] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11846.321517] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11852.511630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11852.925705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11853.027346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11857.444332] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11858.405158] LustreError: 417994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11858.425203] LustreError: 417994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11863.527447] LustreError: 417993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11866.110703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11866.559227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11870.882316] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11877.624023] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11883.264371] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11892.193252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11892.199916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11892.204436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11892.223272] Lustre: Skipped 1 previous similar message [11895.816885] Lustre: server umount lustre-OST0000 complete [11902.437278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11902.440185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11902.442166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11902.442174] Lustre: Skipped 1 previous similar message [11902.460561] Lustre: Skipped 1 previous similar message [11907.553989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11907.565079] Lustre: Skipped 2 previous similar messages [11912.683792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11912.693032] Lustre: Skipped 1 previous similar message [11913.184134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11913.443512] Lustre: server umount lustre-MDT0000 complete [11916.801233] LustreError: 417975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391364 with bad export cookie 15601726883257953587 [11916.809440] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11916.816776] LustreError: 417975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11917.066576] Lustre: server umount lustre-MDT0001 complete [11926.781777] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [11929.219817] Key type lgssc unregistered [11929.510191] LNet: 421536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11929.524982] LNetError: 421536:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11929.567650] LNet: Removed LNI 192.168.203.156@tcp [11930.324651] Key type .llcrypt unregistered [11930.328662] Key type ._llcrypt unregistered [11955.703891] Key type ._llcrypt registered [11955.707807] Key type .llcrypt registered [11955.819665] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [11957.113498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11957.192773] alg: No test for adler32 (adler32-zlib) [11958.290416] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [11958.597968] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [11960.264567] Key type lgssc registered [11961.558703] Lustre: Echo OBD driver; http://www.lustre.org/ [11969.457708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11969.484180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11971.015472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11977.148413] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11980.260126] LustreError: 422953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.295202] LustreError: 422953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11985.379202] LustreError: 422972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11986.778829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11987.284929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11993.103273] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12002.494277] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12009.009857] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12016.045826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12016.117622] LustreError: 424777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12016.307472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12021.774702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12023.347203] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12030.594805] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12038.126125] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12038.133643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12038.151868] Lustre: Skipped 2 previous similar messages [12038.164176] Lustre: Skipped 1 previous similar message [12041.645770] Lustre: server umount lustre-MDT0000 complete [12043.236589] LustreError: 422972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12043.271523] LustreError: 422972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12043.955642] LustreError: 422934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.56@tcp arrived at 1787391491 with bad export cookie 4742422327951489410 [12043.967752] LustreError: 422934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12045.571642] LustreError: 422935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391493 with bad export cookie 4742422327951488521 [12045.578803] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12045.584504] LustreError: 422935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12048.357809] LustreError: 422953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12048.358355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12048.360508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12048.374718] LustreError: 422953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12048.408287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12048.413250] Lustre: Skipped 2 previous similar messages [12050.403418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12051.742766] Lustre: server umount lustre-OST0000 complete [12057.588965] Lustre: server umount lustre-MDT0001 complete [12070.650626] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [12074.059319] Key type lgssc unregistered [12074.375633] LNet: 426638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12074.381724] LNetError: 426638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12074.392074] LNet: Removed LNI 192.168.203.156@tcp [12075.200528] Key type .llcrypt unregistered [12075.209141] Key type ._llcrypt unregistered [12102.061796] Key type ._llcrypt registered [12102.067731] Key type .llcrypt registered [12102.155760] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [12103.142060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12103.234257] alg: No test for adler32 (adler32-zlib) [12104.402290] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [12104.685496] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [12106.352133] Key type lgssc registered [12107.786683] Lustre: Echo OBD driver; http://www.lustre.org/ [12116.161531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12116.223473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12117.822566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12122.178691] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12127.208797] LustreError: 428056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12127.239849] LustreError: 428056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12131.263590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12131.660649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12135.740593] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12143.984409] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12150.388550] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12158.192484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12158.939783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12163.078375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12165.692594] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12173.440880] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12177.893840] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12177.906674] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12177.930505] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12180.295658] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.56@tcp (stopping) [12181.476352] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12181.483859] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12184.145401] Lustre: server umount lustre-MDT0001 complete [12185.421082] LustreError: 428051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12186.607082] LustreError: 428056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12186.639054] LustreError: 428056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12191.716047] LustreError: 428057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12191.785257] LustreError: 428057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12193.764930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12193.779606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12193.795743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12193.801589] Lustre: Skipped 1 previous similar message [12196.835158] LustreError: 428056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12198.879037] Lustre: server umount lustre-OST0000 complete [12205.090723] Lustre: server umount lustre-MDT0000 complete [12223.218472] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [12226.278671] Key type lgssc unregistered [12226.682581] LNet: 431746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12226.695246] LNetError: 431746:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12226.728660] LNet: Removed LNI 192.168.203.156@tcp [12227.420665] Key type .llcrypt unregistered [12227.424923] Key type ._llcrypt unregistered [12255.185751] Key type ._llcrypt registered [12255.188737] Key type .llcrypt registered [12255.322871] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [12271.822151] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [12272.837721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12272.918104] alg: No test for adler32 (adler32-zlib) [12274.049354] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [12274.365620] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [12276.152236] Key type lgssc registered [12277.765272] Lustre: Echo OBD driver; http://www.lustre.org/ [12320.367123] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [12337.404534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12337.440523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12338.878346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12338.977278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12339.169957] Lustre: lustre-MDT0000: new disk, initializing [12339.339467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12339.357224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12344.287321] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12360.415155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12360.725156] Lustre: lustre-OST0000: new disk, initializing [12360.729363] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12360.733244] Lustre: Skipped 1 previous similar message [12360.737447] Lustre: 436781:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12360.785153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12367.618951] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12368.514081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12368.536066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12368.575447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12377.476453] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12396.316938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12396.564299] Lustre: 435763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12396.691321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12396.802331] Lustre: lustre-MDT0001: new disk, initializing [12397.004793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12397.043755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12397.054815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12403.950364] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12413.951726] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12414.218586] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12418.675519] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12418.926534] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12423.651980] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12423.667906] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12423.697132] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12423.697614] Lustre: Skipped 1 previous similar message [12423.718417] Lustre: Skipped 1 previous similar message [12426.103981] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.56@tcp (stopping) [12427.695108] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12427.705212] Lustre: Skipped 1 previous similar message [12430.201715] Lustre: server umount lustre-MDT0001 complete [12436.319299] LustreError: 435770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12436.330051] LustreError: 435770:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12437.986736] LustreError: 435768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12438.033858] LustreError: 435768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12443.124808] LustreError: 435774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12443.167748] LustreError: 435774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12448.226735] LustreError: 435768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12448.246188] LustreError: 435768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12450.277216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12450.284791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12450.308246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12453.347852] LustreError: 435773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12456.254853] Lustre: server umount lustre-OST0000 complete [12466.864954] Lustre: server umount lustre-MDT0000 complete [12483.282892] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [12486.680180] Key type lgssc unregistered [12487.008928] LNet: 440531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12487.019026] LNetError: 440531:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12487.038549] LNet: Removed LNI 192.168.203.156@tcp [12487.994376] Key type .llcrypt unregistered [12487.995918] Key type ._llcrypt unregistered [12530.092759] Key type ._llcrypt registered [12530.095718] Key type .llcrypt registered [12530.210654] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [12530.479455] Key type .llcrypt unregistered [12530.485735] Key type ._llcrypt unregistered [12549.032412] Key type ._llcrypt registered [12549.034563] Key type .llcrypt registered [12549.169873] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [12550.352023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12550.411911] alg: No test for adler32 (adler32-zlib) [12551.446960] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [12551.703991] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [12553.353443] Key type lgssc registered [12554.636927] Lustre: Echo OBD driver; http://www.lustre.org/ [12583.416459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12583.440420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12584.718398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12584.781658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12584.917969] Lustre: lustre-MDT0000: new disk, initializing [12585.019285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12585.033455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12589.824435] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12601.529521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12601.670587] Lustre: 443905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12601.725573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12601.729919] Lustre: Skipped 1 previous similar message [12601.803361] Lustre: lustre-MDT0001: new disk, initializing [12601.905495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12601.960122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12601.977383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12606.239215] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12616.375480] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12622.129345] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12628.297609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12628.655234] Lustre: lustre-OST0000: new disk, initializing [12628.660987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12628.669508] Lustre: 445876:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12628.827941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12634.239814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12634.257805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12634.440395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12635.124178] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12645.087945] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12653.025482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12653.037948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12653.061381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12655.079123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12655.081882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12659.015949] Lustre: server umount lustre-MDT0000 complete [12663.442148] LustreError: 443898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392111 with bad export cookie 16000135923203545131 [12663.446917] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12663.470734] LustreError: 443898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12663.890810] Lustre: server umount lustre-MDT0001 complete [12677.691401] Lustre: server umount lustre-OST0000 complete [12692.898884] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:48:57 (1787392137) [12702.882061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12703.901362] LustreError: 447903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12703.950667] LustreError: 447903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12704.023104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12708.683831] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12709.350413] LustreError: 447904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12709.381994] LustreError: 447904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12713.448218] LustreError: 447903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12716.805959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12717.073393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12721.304745] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12728.696754] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12734.024117] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12752.885863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12753.078986] Lustre: Found index 0 for lustre-OST0000, updating log [12753.254556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12758.234642] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12764.722328] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12774.768593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12782.049467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12782.055897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12782.066875] Lustre: Skipped 1 previous similar message [12782.073548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12782.085151] Lustre: Skipped 2 previous similar messages [12789.222791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12789.229282] Lustre: Skipped 4 previous similar messages [12795.360800] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12795.376022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12795.606039] Lustre: server umount lustre-OST0000 complete [12811.753173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12811.756181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12811.774630] Lustre: Skipped 2 previous similar messages [12811.789944] Lustre: Skipped 2 previous similar messages [12815.008667] LNet: 1 local NIs in recovery (showing 1): 192.168.203.156@tcp [12817.765745] Lustre: server umount lustre-MDT0000 complete [12821.989986] LustreError: 447898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12822.043583] LustreError: 447898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12823.637379] LustreError: 448639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392271 with bad export cookie 16000135923203546524 [12823.658975] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12823.664650] LustreError: 448639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12824.288089] Lustre: server umount lustre-MDT0001 complete [12833.364862] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:51:18 (1787392278) [13452.397174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13452.953810] LustreError: 454351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13453.022191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13457.859112] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13458.402857] LustreError: 454352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13458.417832] LustreError: 454352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13462.503436] LustreError: 454351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13465.626480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13466.090613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13470.307779] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13477.290306] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13482.822124] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13502.652599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13503.044750] Lustre: Found index 0 for lustre-OST0000, updating log [13503.441937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13505.313204] LNet: 1 local NIs in recovery (showing 1): 192.168.203.156@tcp [13510.537934] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13518.691608] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13520.802309] Lustre: server umount lustre-OST0000 complete [13526.013511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13526.021622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13526.041932] Lustre: Skipped 1 previous similar message [13526.051020] Lustre: Skipped 2 previous similar messages [13528.164626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13528.175018] Lustre: Skipped 4 previous similar messages [13532.264144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13532.273728] Lustre: Skipped 6 previous similar messages [13539.298500] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13539.495846] LustreError: 454346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13539.512507] LustreError: 454346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13539.642600] Lustre: server umount lustre-MDT0000 complete [13543.589590] LustreError: 456171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13543.615729] LustreError: 456171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [13543.751050] LustreError: 455086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392991 with bad export cookie 16000135923203547609 [13543.767386] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13543.775560] LustreError: 455086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13544.106722] Lustre: server umount lustre-MDT0001 complete [13565.729282] LNet: 1 local NIs in recovery (showing 1): 192.168.203.156@tcp [13573.871552] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [13583.440338] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [13619.491913] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:04:25 (1787393065) [13637.148685] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [13640.492446] Key type lgssc unregistered [13640.896267] LNet: 460965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13640.903199] LNetError: 460965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13763.815287] LNet: Removed LNI 192.168.203.156@tcp [13764.746857] Key type .llcrypt unregistered [13764.749365] Key type ._llcrypt unregistered [13780.247605] Key type ._llcrypt registered [13780.250989] Key type .llcrypt registered [13780.329996] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [13781.370033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13781.470917] alg: No test for adler32 (adler32-zlib) [13782.569755] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [13782.818051] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [13784.507240] Key type lgssc registered [13785.550413] Lustre: Echo OBD driver; http://www.lustre.org/ [13789.781489] LNet: Added LNI 192.168.203.156@tcp100 [8/256/0/180] [13790.759259] LNet: Added LNI 192.168.203.156@tcp101 [8/256/0/180] [13791.571183] LNet: Added LNI 192.168.203.156@tcp102 [8/256/0/180] [13792.443804] LNet: Added LNI 192.168.203.156@tcp103 [8/256/0/180] [13793.385142] LNet: Added LNI 192.168.203.156@tcp104 [8/256/0/180] [13794.356915] LNet: Added LNI 192.168.203.156@tcp105 [8/256/0/180] [13795.177361] LNet: Added LNI 192.168.203.156@tcp106 [8/256/0/180] [13795.976215] LNet: Added LNI 192.168.203.156@tcp107 [8/256/0/180] [13796.935548] LNet: Added LNI 192.168.203.156@tcp108 [8/256/0/180] [13797.864264] LNet: Added LNI 192.168.203.156@tcp109 [8/256/0/180] [13798.698445] LNet: Added LNI 192.168.203.156@tcp110 [8/256/0/180] [13799.665377] LNet: Added LNI 192.168.203.156@tcp111 [8/256/0/180] [13800.509824] LNet: Added LNI 192.168.203.156@tcp112 [8/256/0/180] [13801.723421] LNet: Added LNI 192.168.203.156@tcp113 [8/256/0/180] [13802.668279] LNet: Added LNI 192.168.203.156@tcp114 [8/256/0/180] [13803.851973] LNet: Added LNI 192.168.203.156@tcp115 [8/256/0/180] [13805.183860] LNet: Added LNI 192.168.203.156@tcp116 [8/256/0/180] [13806.117866] LNet: Added LNI 192.168.203.156@tcp117 [8/256/0/180] [13807.433683] LNet: Added LNI 192.168.203.156@tcp118 [8/256/0/180] [13808.891550] LNet: Added LNI 192.168.203.156@tcp119 [8/256/0/180] [13810.331707] LNet: Added LNI 192.168.203.156@tcp120 [8/256/0/180] [13811.250856] LNet: Added LNI 192.168.203.156@tcp121 [8/256/0/180] [13812.186726] LNet: Added LNI 192.168.203.156@tcp122 [8/256/0/180] [13813.388781] LNet: Added LNI 192.168.203.156@tcp123 [8/256/0/180] [13814.367386] LNet: Added LNI 192.168.203.156@tcp124 [8/256/0/180] [13815.500826] LNet: Added LNI 192.168.203.156@tcp125 [8/256/0/180] [13816.678845] LNet: Added LNI 192.168.203.156@tcp126 [8/256/0/180] [13817.857083] LNet: Added LNI 192.168.203.156@tcp127 [8/256/0/180] [13818.835880] LNet: Added LNI 192.168.203.156@tcp128 [8/256/0/180] [13819.770558] LNet: Added LNI 192.168.203.156@tcp129 [8/256/0/180] [13820.629518] LNet: Added LNI 192.168.203.156@tcp130 [8/256/0/180] [13821.429851] LNet: Added LNI 192.168.203.156@tcp131 [8/256/0/180] [13822.367031] LNet: Added LNI 192.168.203.156@tcp132 [8/256/0/180] [13823.288478] LNet: Added LNI 192.168.203.156@tcp133 [8/256/0/180] [13824.225116] LNet: Added LNI 192.168.203.156@tcp134 [8/256/0/180] [13825.139462] LNet: Added LNI 192.168.203.156@tcp135 [8/256/0/180] [13832.493292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13832.539570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13833.774457] Lustre: 464003:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.156@tcp: can't add NID 192.168.203.156@tcp130: rc = -75 [13833.971358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13834.012812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13834.090977] Lustre: lustre-MDT0000: new disk, initializing [13834.297586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13834.319597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13839.574916] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13850.621901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13850.838987] Lustre: 464020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13850.933736] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13850.937435] Lustre: Skipped 1 previous similar message [13851.006992] Lustre: lustre-MDT0001: new disk, initializing [13851.034447] Lustre: 464822:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.156@tcp: can't add NID 192.168.203.156@tcp130: rc = -75 [13851.043235] Lustre: 464822:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13851.175830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13851.212207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13851.228324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13856.282169] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13860.944388] Lustre: 465248:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.156@tcp: can't add NID 192.168.203.156@tcp130: rc = -75 [13860.956335] Lustre: 465248:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13866.476215] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13871.914338] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13878.420835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13878.691113] Lustre: lustre-OST0000: new disk, initializing [13878.697229] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13878.703439] Lustre: 465994:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13878.788648] Lustre: 465999:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.156@tcp: can't add NID 192.168.203.156@tcp130: rc = -75 [13878.794230] Lustre: 465999:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13878.837708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13883.959791] Lustre: 466292:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.156@tcp: can't add NID 192.168.203.156@tcp130: rc = -75 [13883.974125] Lustre: 466292:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13884.020295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13884.035333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13884.232231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13884.897434] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13894.311418] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13904.870964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13904.871360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13904.878500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13904.886532] Lustre: Skipped 1 previous similar message [13909.985867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13909.992706] Lustre: Skipped 1 previous similar message [13915.114195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13915.119030] Lustre: Skipped 1 previous similar message [13916.133911] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13916.422797] Lustre: server umount lustre-OST0000 complete [13920.226416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13920.256903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13920.257681] Lustre: Skipped 1 previous similar message [13920.266370] Lustre: Skipped 2 previous similar messages [13925.348539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13925.357041] Lustre: Skipped 1 previous similar message [13926.093635] Lustre: server umount lustre-MDT0000 complete [13929.624712] LustreError: 464012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393377 with bad export cookie 5571658253158646691 [13929.638061] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13929.643374] LustreError: 464012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13930.168014] Lustre: server umount lustre-MDT0001 complete [13940.665499] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [13943.599937] Key type lgssc unregistered [13943.973932] LNet: 467908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13943.985872] LNetError: 467908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13943.994203] LNet: Removed LNI 192.168.203.156@tcp [13945.056180] LNet: Removed LNI 192.168.203.156@tcp100 [13945.060804] LNetError: 467908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13945.066029] LNetError: 467908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13946.081231] LNet: Removed LNI 192.168.203.156@tcp101 [13946.086476] LNetError: 467908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13946.096090] LNet: Removed LNI 192.168.203.156@tcp102 [13947.168721] LNet: Removed LNI 192.168.203.156@tcp103 [13948.192175] LNet: Removed LNI 192.168.203.156@tcp104 [13948.198704] LNetError: 467908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13948.210576] LNetError: 467908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13948.227747] LNet: Removed LNI 192.168.203.156@tcp105 [13949.280750] LNet: Removed LNI 192.168.203.156@tcp106 [13950.305717] LNet: Removed LNI 192.168.203.156@tcp107 [13951.328128] LNet: Removed LNI 192.168.203.156@tcp108 [13952.354802] LNet: Removed LNI 192.168.203.156@tcp109 [13952.360203] LNetError: 467908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13952.370451] LNetError: 467908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13952.400561] LNet: Removed LNI 192.168.203.156@tcp110 [13953.440226] LNet: Removed LNI 192.168.203.156@tcp111 [13954.464225] LNet: Removed LNI 192.168.203.156@tcp112 [13955.488709] LNet: Removed LNI 192.168.203.156@tcp113 [13956.512344] LNet: Removed LNI 192.168.203.156@tcp114 [13956.518128] LNet: Removed LNI 192.168.203.156@tcp115 [13956.524172] LNet: Removed LNI 192.168.203.156@tcp116 [13956.527698] LNet: Removed LNI 192.168.203.156@tcp117 [13956.539494] LNet: Removed LNI 192.168.203.156@tcp118 [13956.543069] LNet: Removed LNI 192.168.203.156@tcp119 [13956.549889] LNet: Removed LNI 192.168.203.156@tcp120 [13956.555652] LNet: Removed LNI 192.168.203.156@tcp121 [13956.564741] LNet: Removed LNI 192.168.203.156@tcp122 [13956.568715] LNet: Removed LNI 192.168.203.156@tcp123 [13956.573784] LNet: Removed LNI 192.168.203.156@tcp124 [13956.581413] LNet: Removed LNI 192.168.203.156@tcp125 [13956.593554] LNet: Removed LNI 192.168.203.156@tcp126 [13956.601912] LNet: Removed LNI 192.168.203.156@tcp127 [13956.616671] LNet: Removed LNI 192.168.203.156@tcp128 [13956.623725] LNet: Removed LNI 192.168.203.156@tcp129 [13956.627256] LNet: Removed LNI 192.168.203.156@tcp130 [13956.630846] LNet: Removed LNI 192.168.203.156@tcp131 [13956.635502] LNet: Removed LNI 192.168.203.156@tcp132 [13956.640987] LNet: Removed LNI 192.168.203.156@tcp133 [13956.646123] LNet: Removed LNI 192.168.203.156@tcp134 [13956.657915] LNet: Removed LNI 192.168.203.156@tcp135 [13957.837954] Key type .llcrypt unregistered [13957.843205] Key type ._llcrypt unregistered [13982.270559] Key type ._llcrypt registered [13982.275802] Key type .llcrypt registered [13982.406540] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [13998.382452] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [13999.434834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13999.456634] alg: No test for adler32 (adler32-zlib) [14000.515706] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [14000.696460] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [14002.417071] Key type lgssc registered [14003.664033] Lustre: Echo OBD driver; http://www.lustre.org/ [14038.475359] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:11:23 (1787393483) [14055.393403] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [14058.177083] Key type lgssc unregistered [14058.510847] LNet: 471994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14058.519241] LNetError: 471994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14058.553915] LNet: Removed LNI 192.168.203.156@tcp [14059.671557] Key type .llcrypt unregistered [14059.676281] Key type ._llcrypt unregistered [14075.787950] Key type ._llcrypt registered [14075.790810] Key type .llcrypt registered [14075.903100] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [14077.007905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14077.017646] alg: No test for adler32 (adler32-zlib) [14078.121117] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [14078.447473] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [14080.216264] Key type lgssc registered [14081.847376] Lustre: Echo OBD driver; http://www.lustre.org/ [14094.284950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14094.388397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14095.656260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14095.699690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14095.783915] Lustre: lustre-MDT0000: new disk, initializing [14095.929618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14095.943817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14100.504579] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14111.497348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14111.658774] Lustre: 473207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14111.746210] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14111.770779] Lustre: Skipped 1 previous similar message [14111.870910] Lustre: lustre-MDT0001: new disk, initializing [14111.976831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14112.033829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14112.054192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14116.278947] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14125.887170] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14131.826978] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14138.380734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14138.714437] Lustre: lustre-OST0000: new disk, initializing [14138.718239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14138.728146] Lustre: 475177:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14138.813169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14145.062201] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14146.666760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14146.682610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14146.782400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14154.774267] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14158.747731] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing lnet_if_list [14163.793935] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14163.821854] LNet: Added LNI 192.168.203.156@tcp10 [8/256/0/180] [14169.323871] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14169.350914] LNet: Added LNI 192.168.203.156@tcp11 [8/256/0/180] [14173.517841] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14173.553472] LNet: Added LNI 192.168.203.156@tcp12 [8/256/0/180] [14178.098619] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14178.136543] LNet: Added LNI 192.168.203.156@tcp13 [8/256/0/180] [14183.192639] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14183.238586] LNet: Added LNI 192.168.203.156@tcp14 [8/256/0/180] [14187.743723] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14187.776516] LNet: Added LNI 192.168.203.156@tcp15 [8/256/0/180] [14192.840890] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14192.866430] LNet: Added LNI 192.168.203.156@tcp16 [8/256/0/180] [14198.296762] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14198.322431] LNet: Added LNI 192.168.203.156@tcp17 [8/256/0/180] [14202.932661] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14202.955799] LNet: Added LNI 192.168.203.156@tcp18 [8/256/0/180] [14208.127103] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14208.166422] LNet: Added LNI 192.168.203.156@tcp19 [8/256/0/180] [14212.345453] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14212.381058] LNet: Added LNI 192.168.203.156@tcp20 [8/256/0/180] [14217.204134] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14217.231647] LNet: Added LNI 192.168.203.156@tcp21 [8/256/0/180] [14222.227112] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14222.244574] LNet: Added LNI 192.168.203.156@tcp22 [8/256/0/180] [14227.439590] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14227.468360] LNet: Added LNI 192.168.203.156@tcp23 [8/256/0/180] [14234.152831] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14234.193461] LNet: Added LNI 192.168.203.156@tcp24 [8/256/0/180] [14241.311915] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14241.366427] LNet: Added LNI 192.168.203.156@tcp25 [8/256/0/180] [14248.143839] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14248.164439] LNet: Added LNI 192.168.203.156@tcp26 [8/256/0/180] [14254.871354] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14254.903582] LNet: Added LNI 192.168.203.156@tcp27 [8/256/0/180] [14261.087737] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14261.124807] LNet: Added LNI 192.168.203.156@tcp28 [8/256/0/180] [14267.749315] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14267.779988] LNet: Added LNI 192.168.203.156@tcp29 [8/256/0/180] [14274.734958] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14274.775548] LNet: Added LNI 192.168.203.156@tcp30 [8/256/0/180] [14280.585467] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14280.613811] LNet: Added LNI 192.168.203.156@tcp31 [8/256/0/180] [14286.339460] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14286.365926] LNet: Added LNI 192.168.203.156@tcp32 [8/256/0/180] [14291.905679] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14291.936380] LNet: Added LNI 192.168.203.156@tcp33 [8/256/0/180] [14297.677237] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14297.697185] LNet: Added LNI 192.168.203.156@tcp34 [8/256/0/180] [14303.797797] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14303.863440] LNet: Added LNI 192.168.203.156@tcp35 [8/256/0/180] [14309.428616] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14309.450607] LNet: Added LNI 192.168.203.156@tcp36 [8/256/0/180] [14315.987691] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14316.017970] LNet: Added LNI 192.168.203.156@tcp37 [8/256/0/180] [14322.710574] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14322.744459] LNet: Added LNI 192.168.203.156@tcp38 [8/256/0/180] [14330.097830] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14330.137959] LNet: Added LNI 192.168.203.156@tcp39 [8/256/0/180] [14336.648884] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14336.683249] LNet: Added LNI 192.168.203.156@tcp40 [8/256/0/180] [14343.038295] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14343.069898] LNet: Added LNI 192.168.203.156@tcp41 [8/256/0/180] [14348.998492] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14349.036211] LNet: Added LNI 192.168.203.156@tcp42 [8/256/0/180] [14355.474536] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14355.502311] LNet: Added LNI 192.168.203.156@tcp43 [8/256/0/180] [14362.589219] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14362.625662] LNet: Added LNI 192.168.203.156@tcp44 [8/256/0/180] [14370.172596] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14370.219436] LNet: Added LNI 192.168.203.156@tcp45 [8/256/0/180] [14377.534175] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14377.563805] LNet: Added LNI 192.168.203.156@tcp46 [8/256/0/180] [14385.191106] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14385.233446] LNet: Added LNI 192.168.203.156@tcp47 [8/256/0/180] [14392.845482] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14392.873964] LNet: Added LNI 192.168.203.156@tcp48 [8/256/0/180] [14401.138342] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14401.178079] LNet: Added LNI 192.168.203.156@tcp49 [8/256/0/180] [14409.447192] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14409.495035] LNet: Added LNI 192.168.203.156@tcp50 [8/256/0/180] [14429.667297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14429.679866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14429.694794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14431.721217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14431.722507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14431.743377] Lustre: Skipped 1 previous similar message [14431.760351] Lustre: Skipped 2 previous similar messages [14433.471381] Lustre: server umount lustre-MDT0000 complete [14437.972040] LustreError: 473201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393885 with bad export cookie 5821557659162177098 [14437.980763] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14437.990861] LustreError: 473201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14438.368635] Lustre: server umount lustre-MDT0001 complete [14456.001962] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [14457.335951] Lustre: 472477:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787393889/real 1787393889] req@000000000a2179b3 x1874217954084096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787393905 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14457.365095] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14458.400383] Lustre: 472477:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787393890/real 1787393890] req@000000007f46afbc x1874217954084224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787393906 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14462.432464] Lustre: 472480:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787393894/real 1787393894] req@00000000d8aa55f1 x1874217954084352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787393910 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14464.688943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14469.603227] Lustre: 472478:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787393901/real 1787393901] req@000000001d115148 x1874217954084736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787393917 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14469.635388] Lustre: 472478:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14482.979283] LustreError: 472476:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004d94f63e x1874217954085376/t0(0) o250->MGC192.168.203.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14483.315824] LustreError: 475171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14483.336855] LustreError: 475171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14483.390671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14486.447161] LustreError: 475172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14486.454682] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14487.458889] LustreError: 475172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14487.728340] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14492.643444] LustreError: 475171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14496.669920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14497.023843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14501.809475] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14503.416651] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14509.765336] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14514.864313] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14537.187524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14537.190148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14537.205472] Lustre: Skipped 1 previous similar message [14540.951657] Lustre: server umount lustre-OST0000 complete [14547.425630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14547.431073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14547.442064] Lustre: Skipped 1 previous similar message [14547.456199] Lustre: Skipped 2 previous similar messages [14551.076708] Lustre: server umount lustre-MDT0000 complete [14552.546379] LustreError: 499420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14552.568334] LustreError: 499420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14554.725973] LustreError: 473200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394002 with bad export cookie 5821557659162179625 [14554.732424] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14554.736629] LustreError: 473200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14555.109123] Lustre: server umount lustre-MDT0001 complete [14565.424746] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [14568.397921] Key type lgssc unregistered [14568.817302] LNet: 502216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14568.828727] LNetError: 502216:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14568.841569] LNet: Removed LNI 192.168.203.156@tcp [14568.846888] LNet: Removed LNI 192.168.203.156@tcp10 [14568.856788] LNet: Removed LNI 192.168.203.156@tcp11 [14568.861427] LNet: Removed LNI 192.168.203.156@tcp12 [14568.864609] LNet: Removed LNI 192.168.203.156@tcp13 [14568.867900] LNet: Removed LNI 192.168.203.156@tcp14 [14568.870406] LNet: Removed LNI 192.168.203.156@tcp15 [14568.872347] LNet: Removed LNI 192.168.203.156@tcp16 [14568.875460] LNet: Removed LNI 192.168.203.156@tcp17 [14568.878783] LNet: Removed LNI 192.168.203.156@tcp18 [14568.881869] LNet: Removed LNI 192.168.203.156@tcp19 [14568.884547] LNet: Removed LNI 192.168.203.156@tcp20 [14568.888381] LNet: Removed LNI 192.168.203.156@tcp21 [14568.890924] LNet: Removed LNI 192.168.203.156@tcp22 [14568.894226] LNet: Removed LNI 192.168.203.156@tcp23 [14568.897265] LNet: Removed LNI 192.168.203.156@tcp24 [14568.899811] LNet: Removed LNI 192.168.203.156@tcp25 [14568.902597] LNet: Removed LNI 192.168.203.156@tcp26 [14568.905594] LNet: Removed LNI 192.168.203.156@tcp27 [14568.909399] LNet: Removed LNI 192.168.203.156@tcp28 [14568.913359] LNet: Removed LNI 192.168.203.156@tcp29 [14568.916884] LNet: Removed LNI 192.168.203.156@tcp30 [14568.921562] LNet: Removed LNI 192.168.203.156@tcp31 [14568.927683] LNet: Removed LNI 192.168.203.156@tcp32 [14568.933606] LNet: Removed LNI 192.168.203.156@tcp33 [14568.938762] LNet: Removed LNI 192.168.203.156@tcp34 [14568.943206] LNet: Removed LNI 192.168.203.156@tcp35 [14568.947330] LNet: Removed LNI 192.168.203.156@tcp36 [14568.952281] LNet: Removed LNI 192.168.203.156@tcp37 [14568.963089] LNet: Removed LNI 192.168.203.156@tcp38 [14568.965809] LNet: Removed LNI 192.168.203.156@tcp39 [14568.968678] LNet: Removed LNI 192.168.203.156@tcp40 [14568.974493] LNet: Removed LNI 192.168.203.156@tcp41 [14568.978823] LNet: Removed LNI 192.168.203.156@tcp42 [14568.981460] LNet: Removed LNI 192.168.203.156@tcp43 [14568.984440] LNet: Removed LNI 192.168.203.156@tcp44 [14568.987035] LNet: Removed LNI 192.168.203.156@tcp45 [14568.989913] LNet: Removed LNI 192.168.203.156@tcp46 [14568.992587] LNet: Removed LNI 192.168.203.156@tcp47 [14568.999092] LNet: Removed LNI 192.168.203.156@tcp48 [14569.002389] LNet: Removed LNI 192.168.203.156@tcp49 [14569.015404] LNet: Removed LNI 192.168.203.156@tcp50 [14569.725616] Key type .llcrypt unregistered [14569.727569] Key type ._llcrypt unregistered [14592.272395] Key type ._llcrypt registered [14592.274246] Key type .llcrypt registered [14592.537807] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [14610.584288] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [14611.829866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14611.969310] alg: No test for adler32 (adler32-zlib) [14613.219467] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [14613.464702] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [14615.160193] Key type lgssc registered [14616.441282] Lustre: Echo OBD driver; http://www.lustre.org/ [14652.971034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14653.002189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14654.236162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14654.291736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14654.416694] Lustre: lustre-MDT0000: new disk, initializing [14654.510901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14654.534982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14658.952388] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14669.266097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14669.450365] Lustre: 505414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14669.536330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14669.544467] Lustre: Skipped 1 previous similar message [14669.657689] Lustre: lustre-MDT0001: new disk, initializing [14669.782928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14669.828536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14669.847100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14674.344260] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14683.865659] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14689.424753] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14696.652142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14696.806848] Lustre: *** cfs_fail_loc=90f, val=0*** [14696.912893] Lustre: lustre-OST0000: new disk, initializing [14696.916827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14696.923437] Lustre: 507429:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14697.007506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14702.696880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14702.713446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14702.892914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14702.908796] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14711.922918] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14723.568965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14723.588140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14723.589698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14723.606322] Lustre: Skipped 1 previous similar message [14728.679393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14728.692740] Lustre: Skipped 1 previous similar message [14733.280117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14733.479538] Lustre: server umount lustre-OST0000 complete [14738.916190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14738.940156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14738.946975] Lustre: Skipped 2 previous similar messages [14738.987664] Lustre: Skipped 1 previous similar message [14744.036272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14744.044278] Lustre: Skipped 1 previous similar message [14744.447306] Lustre: server umount lustre-MDT0000 complete [14749.533439] LustreError: 505407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394197 with bad export cookie 7270081722583032662 [14749.537563] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14749.542691] LustreError: 505407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14750.117131] Lustre: server umount lustre-MDT0001 complete [14761.952751] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [14765.110850] Key type lgssc unregistered [14765.387858] LNet: 509337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14765.406776] LNetError: 509337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14765.429913] LNet: Removed LNI 192.168.203.156@tcp [14766.250277] Key type .llcrypt unregistered [14766.253506] Key type ._llcrypt unregistered [14786.438703] Key type ._llcrypt registered [14786.441865] Key type .llcrypt registered [14786.524620] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [14787.653723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14787.715124] alg: No test for adler32 (adler32-zlib) [14788.742371] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [14788.970116] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [14790.664473] Key type lgssc registered [14791.908814] Lustre: Echo OBD driver; http://www.lustre.org/ [14800.220646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14800.241456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14801.787829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14806.510422] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14812.131688] LustreError: 510609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14812.149649] LustreError: 510609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14815.055670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14815.505544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14819.999417] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14826.985113] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14832.551328] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14839.565473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14839.928175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14846.090632] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14852.938221] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14862.878134] loop0: detected capacity change from 0 to 204800000 [14863.025032] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14863.034834] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14863.051422] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14868.755471] loop0: detected capacity change from 0 to 204800000 [14868.998179] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14869.008480] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14869.026438] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14874.227929] loop0: detected capacity change from 0 to 204800000 [14879.436130] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14879.632810] Lustre: fs15246-OST0000: new disk, initializing [14879.637831] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14879.648989] Lustre: 514202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14879.742612] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14886.434674] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14895.483494] loop1: detected capacity change from 0 to 204800000 [14900.380551] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14900.505422] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14900.587931] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14900.670559] Lustre: fs15246-MDT0000: new disk, initializing [14900.823714] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14900.854639] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14900.978319] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14902.049252] Lustre: 515327:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000009f5b24ec x1874218698454912/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787394365 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14905.259034] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14907.124868] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14907.142380] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14907.205906] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14931.959270] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14931.987726] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14934.329766] Lustre: server umount fs15246-MDT0000 complete [14937.064833] LustreError: 514368:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14942.181708] LustreError: 510603:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14943.763449] Lustre: server umount fs15246-OST0000 complete [14954.989087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14954.998118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14955.016446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14957.541886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14957.548274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14961.045700] Lustre: server umount lustre-OST0000 complete [14967.785147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14967.806573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14967.819891] Lustre: Skipped 2 previous similar messages [14971.862658] Lustre: server umount lustre-MDT0000 complete [14972.906359] LustreError: 510608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14972.946590] LustreError: 510608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14975.534424] LustreError: 510588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394423 with bad export cookie 9564980297170025807 [14975.538598] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14975.550921] LustreError: 510588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14975.948336] Lustre: server umount lustre-MDT0001 complete [14986.954825] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [14989.861527] Key type lgssc unregistered [14990.117589] LNet: 518420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14990.125397] LNetError: 518420:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14990.148868] LNet: Removed LNI 192.168.203.156@tcp [14990.959913] Key type .llcrypt unregistered [14990.964332] Key type ._llcrypt unregistered [15009.610504] Key type ._llcrypt registered [15009.612306] Key type .llcrypt registered [15009.743221] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [15010.756345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15010.785939] alg: No test for adler32 (adler32-zlib) [15011.991221] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [15012.256513] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [15013.944203] Key type lgssc registered [15014.989356] Lustre: Echo OBD driver; http://www.lustre.org/ [15054.053768] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [15064.623416] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [15099.117121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15099.135832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15100.439706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15100.490122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15100.598782] Lustre: lustre-MDT0000: new disk, initializing [15100.686144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15100.712062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15105.085250] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15116.436817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15116.554060] Lustre: 522889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15116.613148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15116.617021] Lustre: Skipped 1 previous similar message [15116.686198] Lustre: lustre-MDT0001: new disk, initializing [15116.769822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15116.811086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15116.833515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15121.293306] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15129.903405] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15135.702612] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15142.736952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15143.084032] Lustre: lustre-OST0000: new disk, initializing [15143.094814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15143.112452] Lustre: 524856:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15143.200445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15148.588466] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15149.092537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15149.106815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15149.278915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15157.306147] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15161.884451] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15162.150354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15167.444476] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15167.679719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15170.023860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15170.025701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15170.043081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15170.051910] Lustre: Skipped 1 previous similar message [15175.145475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15175.158400] Lustre: Skipped 2 previous similar messages [15175.800575] Lustre: server umount lustre-OST0000 complete [15180.259143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15180.273930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15180.281593] Lustre: Skipped 1 previous similar message [15185.380714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15185.392863] Lustre: Skipped 1 previous similar message [15185.690652] Lustre: server umount lustre-MDT0000 complete [15189.286134] LustreError: 522882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394637 with bad export cookie 11456944931579895363 [15189.299059] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15189.305400] LustreError: 522882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15189.821127] Lustre: server umount lustre-MDT0001 complete [15216.704752] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [15232.729492] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [15266.300314] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14981 sec ========= 06:31:51 (1787394711) [15268.210161] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:31:53 (1787394713) === [15271.724992] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:31:57 (1787394717) === [15316.463968] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [15319.869553] Key type lgssc unregistered [15320.252689] LNet: 531523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15320.273948] LNetError: 531523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15320.307753] LNet: Removed LNI 192.168.203.156@tcp [15321.567625] Key type .llcrypt unregistered [15321.578126] Key type ._llcrypt unregistered