[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 509753992 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003177] x2apic enabled [ 0.004000] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007259] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010136] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012042] SELinux: Initializing. [ 0.013000] *** VALIDATE selinux *** [ 0.020725] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025255] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030046] *** VALIDATE tmpfs *** [ 0.031489] *** VALIDATE proc *** [ 0.033275] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.036023] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039036] Spectre V2 : User space: Vulnerable [ 0.040012] Speculative Store Bypass: Vulnerable [ 0.042785] debug: unmapping init [mem 0xffffffff8ba59000-0xffffffff8ba60fff] [ 0.044190] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045703] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046027] ... version: 2 [ 0.047017] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049017] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051019] ... fixed-purpose events: 3 [ 0.052015] ... event mask: 000000070000000f [ 0.053376] rcu: Hierarchical SRCU implementation. [ 0.055482] smp: Bringing up secondary CPUs ... [ 0.056619] x86: Booting SMP configuration: [ 0.057030] .... node #0, CPUs: #1 #2 #3 [ 0.060736] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139789] node 0 deferred pages initialised in 74ms [ 0.143315] devtmpfs: initialized [ 0.145290] x86/mm: Memory block size: 128MB [ 0.149219] gcov: version magic: 0x41383552 [ 0.151296] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157339] pinctrl core: initialized pinctrl subsystem [ 0.159216] [ 0.159748] ************************************************************* [ 0.162013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165014] ** ** [ 0.167013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170013] ** ** [ 0.172014] ** This means that this kernel is built to expose internal ** [ 0.174012] ** IOMMU data structures, which may compromise security on ** [ 0.177014] ** your system. ** [ 0.179012] ** ** [ 0.181017] ** If you see this message and you are not debugging the ** [ 0.184019] ** kernel, report this immediately to your vendor! ** [ 0.186012] ** ** [ 0.188019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.190014] ************************************************************* [ 0.193507] NET: Registered protocol family 16 [ 0.195375] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203031] cpuidle: using governor menu [ 0.204748] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205458] PCI: Using configuration type 1 for base access [ 0.206140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.213055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.219049] cryptd: max_cpu_qlen set to 1000 [ 0.220198] ACPI: Added _OSI(Module Device) [ 0.222012] ACPI: Added _OSI(Processor Device) [ 0.223012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.224015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.229261] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235406] ACPI: Interpreter enabled [ 0.237066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238012] ACPI: Using IOAPIC for interrupt routing [ 0.240125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243384] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.252000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.255043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.258021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.261081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.265394] acpiphp: Slot [2] registered [ 0.267144] acpiphp: Slot [5] registered [ 0.268117] acpiphp: Slot [6] registered [ 0.269090] acpiphp: Slot [7] registered [ 0.271062] acpiphp: Slot [8] registered [ 0.272091] acpiphp: Slot [9] registered [ 0.273140] acpiphp: Slot [10] registered [ 0.274079] acpiphp: Slot [3] registered [ 0.275087] acpiphp: Slot [4] registered [ 0.277093] acpiphp: Slot [11] registered [ 0.278109] acpiphp: Slot [12] registered [ 0.280103] acpiphp: Slot [13] registered [ 0.282110] acpiphp: Slot [14] registered [ 0.283068] acpiphp: Slot [15] registered [ 0.284134] acpiphp: Slot [16] registered [ 0.285103] acpiphp: Slot [17] registered [ 0.287138] acpiphp: Slot [18] registered [ 0.288087] acpiphp: Slot [19] registered [ 0.290095] acpiphp: Slot [20] registered [ 0.292101] acpiphp: Slot [21] registered [ 0.293108] acpiphp: Slot [22] registered [ 0.294099] acpiphp: Slot [23] registered [ 0.296124] acpiphp: Slot [24] registered [ 0.298119] acpiphp: Slot [25] registered [ 0.299205] acpiphp: Slot [26] registered [ 0.301132] acpiphp: Slot [27] registered [ 0.303137] acpiphp: Slot [28] registered [ 0.304169] acpiphp: Slot [29] registered [ 0.306075] acpiphp: Slot [30] registered [ 0.307000] acpiphp: Slot [31] registered [ 0.307000] PCI host bridge to bus 0000:00 [ 0.309020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.311020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.313026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.316025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.319027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.321036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.323236] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.326047] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.329423] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.339850] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.345468] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.347015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.350015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.351487] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.353594] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.356045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.358462] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.375018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.379988] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.384485] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.392017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.412055] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.421654] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.428019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.434018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.455017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.466139] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.472018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.479018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.491019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.500669] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.506018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.514016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.532018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.540053] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.547015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.558017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.575019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.591248] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.600018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.610019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.625020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.635537] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.637258] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.638324] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.640286] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.641169] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.645089] iommu: Default domain type: Passthrough [ 0.647312] SCSI subsystem initialized [ 0.648108] ACPI: bus type USB registered [ 0.649172] usbcore: registered new interface driver usbfs [ 0.650081] usbcore: registered new interface driver hub [ 0.651052] usbcore: registered new device driver usb [ 0.652114] pps_core: LinuxPPS API ver. 1 registered [ 0.654008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.655054] PTP clock support registered [ 0.657121] EDAC MC: Ver: 3.0.0 [ 0.659122] PCI: Using ACPI for IRQ routing [ 0.661011] NetLabel: Initializing [ 0.662011] NetLabel: domain hash size = 128 [ 0.664013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.666081] NetLabel: unlabeled traffic allowed by default [ 0.668143] vgaarb: loaded [ 0.670495] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.672015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.683853] clocksource: Switched to clocksource kvm-clock [ 0.777616] VFS: Disk quotas dquot_6.6.0 [ 0.778697] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.780833] *** VALIDATE ramfs *** [ 0.781678] *** VALIDATE hugetlbfs *** [ 0.782609] pnp: PnP ACPI init [ 0.784492] pnp: PnP ACPI: found 6 devices [ 0.807918] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.810921] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.812930] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.814850] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.817389] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.819274] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.821584] NET: Registered protocol family 2 [ 0.823887] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.827598] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.830926] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.834699] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.837141] TCP: Hash tables configured (established 65536 bind 65536) [ 0.839121] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.840911] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.842798] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.844563] NET: Registered protocol family 1 [ 0.847067] RPC: Registered named UNIX socket transport module. [ 0.848365] RPC: Registered udp transport module. [ 0.849325] RPC: Registered tcp transport module. [ 0.850277] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.851482] NET: Registered protocol family 44 [ 0.852456] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.853633] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.854846] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.856285] PCI: CLS 0 bytes, default 64 [ 0.857246] Unpacking initramfs... [ 2.249476] debug: unmapping init [mem 0xffff93b0bcc54000-0xffff93b0bffbffff] [ 2.253221] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.255419] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.257872] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.736054] Initialise system trusted keyrings [ 2.737316] Key type blacklist registered [ 2.738615] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.745236] zbud: loaded [ 2.748058] *** VALIDATE nfs *** [ 2.749034] *** VALIDATE nfs4 *** [ 2.750172] pstore: using deflate compression [ 2.753113] Platform Keyring initialized [ 2.846767] NET: Registered protocol family 38 [ 2.847761] Key type asymmetric registered [ 2.849257] Asymmetric key parser 'x509' registered [ 2.850434] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.852204] io scheduler mq-deadline registered [ 2.853830] io scheduler kyber registered [ 2.855530] io scheduler bfq registered [ 2.857411] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.860088] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.862445] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.864615] ACPI: Power Button [PWRF] [ 2.868359] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.872920] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.887536] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.895195] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.916110] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.943126] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.969209] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.973110] Non-volatile memory driver v1.3 [ 2.974494] Linux agpgart interface v0.103 [ 3.003803] virtio_blk virtio1: [vda] 149944 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.006483] vda: detected capacity change from 0 to 76771328 [ 3.025932] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.028118] vdb: detected capacity change from 0 to 1073741824 [ 3.047820] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.049805] vdc: detected capacity change from 0 to 2621440000 [ 3.063846] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.065600] vdd: detected capacity change from 0 to 2621440000 [ 3.079119] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.081553] vde: detected capacity change from 0 to 4294967296 [ 3.100100] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.101764] vdf: detected capacity change from 0 to 4294967296 [ 3.111846] libphy: Fixed MDIO Bus: probed [ 3.122932] usbcore: registered new interface driver usbserial_generic [ 3.125311] usbserial: USB Serial support registered for generic [ 3.127345] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.131468] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.133169] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.135432] mousedev: PS/2 mouse device common for all mice [ 3.138558] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.142180] rtc_cmos 00:05: RTC can wake from S4 [ 3.145682] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.146084] rtc_cmos 00:05: registered as rtc0 [ 3.149816] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.150105] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.154464] intel_pstate: CPU model not supported [ 3.156715] hid: raw HID events driver (C) Jiri Kosina [ 3.158028] usbcore: registered new interface driver usbhid [ 3.159101] usbhid: USB HID core driver [ 3.160193] drop_monitor: Initializing network drop monitor service [ 3.161773] Initializing XFRM netlink socket [ 3.163105] NET: Registered protocol family 10 [ 3.165381] Segment Routing with IPv6 [ 3.166346] NET: Registered protocol family 17 [ 3.167645] mpls_gso: MPLS GSO support [ 3.173100] RAS: Correctable Errors collector initialized. [ 3.175172] AVX version of gcm_enc/dec engaged. [ 3.177057] AES CTR mode by8 optimization enabled [ 3.250218] sched_clock: Marking stable (3250201773, 0)->(4169663761, -919461988) [ 3.252964] registered taskstats version 1 [ 3.255313] Loading compiled-in X.509 certificates [ 3.257390] zswap: loaded using pool lzo/zbud [ 3.283176] Key type big_key registered [ 3.301675] Key type encrypted registered [ 3.303521] ima: No TPM chip found, activating TPM-bypass! [ 3.305725] ima: Allocated hash algorithm: sha1 [ 3.307552] ima: No architecture policies found [ 3.309276] evm: Initialising EVM extended attributes: [ 3.311115] evm: security.selinux [ 3.312406] evm: security.ima [ 3.313534] evm: security.capability [ 3.314911] evm: HMAC attrs: 0x1 [ 3.317636] rtc_cmos 00:05: setting system clock to 2026-09-13 12:21:08 UTC (1789302068) [ 3.324768] debug: unmapping init [mem 0xffffffff8ca03000-0xffffffff8cbfffff] [ 3.327809] debug: unmapping init [mem 0xffffffff8b782000-0xffffffff8ba58fff] [ 3.338297] Write protecting the kernel read-only data: 28672k [ 3.342063] debug: unmapping init [mem 0xffffffff89e03000-0xffffffff89ffffff] [ 3.344986] debug: unmapping init [mem 0xffffffff8a714000-0xffffffff8a7fffff] [ 3.378690] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.386714] systemd[1]: Detected virtualization kvm. [ 3.388502] systemd[1]: Detected architecture x86-64. [ 3.390452] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.414424] systemd[1]: No hostname configured. [ 3.416156] systemd[1]: Set hostname to . [ 3.418174] random: systemd: uninitialized urandom read (16 bytes read) [ 3.420459] systemd[1]: Initializing machine ID from random generator. [ 3.465336] random: ln: uninitialized urandom read (6 bytes read) [ 3.557229] random: systemd: uninitialized urandom read (16 bytes read) [ 3.559862] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.564988] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.569248] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.130805] device-mapper: uevent: version 1.0.3 [ 4.132682] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] [ 4.425900] random: fast init done Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.712364] virtio_net virtio0 ens2: renamed from eth0 [ 4.753324] scsi host0: ata_piix [ 4.764198] scsi host1: ata_piix [ 4.766479] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.768952] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.859032] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.819854] random: crng init done [ 9.821434] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.029272] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.101715] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.352169] SELinux: Disabled at runtime. [ 11.426477] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.442261] systemd[1]: Detected virtualization kvm. [ 11.445949] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.917639] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.921205] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.926780] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.930728] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.934109] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.942290] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.951323] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 12.109848] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.400450] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.703538] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.789280] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.887874] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.903805] EDAC sbridge: Ver: 1.1.2 [ 14.294472] Key type dns_resolver registered [ 14.568099] NFS: Registering the id_resolver key type [ 14.570063] Key type id_resolver registered [ 14.571332] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg211-server login: [ 39.180396] libcfs: loading out-of-tree module taints kernel. [ 39.200733] Key type ._llcrypt registered [ 39.202154] Key type .llcrypt registered [ 39.254884] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 60.474118] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 62.732136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.754534] alg: No test for adler32 (adler32-zlib) [ 64.523700] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 65.367494] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 67.096478] Key type lgssc registered [ 67.715199] hrtimer: interrupt took 4190562 ns [ 69.306252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.265308] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 134.159852] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 148.198218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 148.213374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.450981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 149.473827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 149.549372] Lustre: lustre-MDT0000: new disk, initializing [ 149.614066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 149.631970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 153.980679] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.092505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.163356] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.197213] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.200033] Lustre: Skipped 1 previous similar message [ 169.289945] Lustre: lustre-MDT0001: new disk, initializing [ 169.377243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 169.408327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 169.410816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.114771] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.171152] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 188.626058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 188.886333] Lustre: lustre-OST0000: new disk, initializing [ 188.890720] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 188.896940] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 188.957104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.474488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.486704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 191.593672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.701386] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.514985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.761128] Lustre: lustre-OST0001: new disk, initializing [ 209.773957] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.784390] Lustre: 9519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.889082] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.994385] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.123116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 217.144531] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 217.232156] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 229.596665] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 237.502262] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 245.962129] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing check_logdir /tmp/testlogs/ [ 250.853541] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing yml_node [ 257.499366] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 261.613810] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 265.096382] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 267.245559] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 13 08:25:30 EDT 2026 [ 288.640728] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 290.195534] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 307.685033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 307.694390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.702404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.233200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.242848] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.256764] Lustre: Skipped 2 previous similar messages [ 311.810128] Lustre: server umount lustre-MDT0000 complete [ 315.245245] LustreError: 8457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302380 with bad export cookie 3990272644285396012 [ 315.255034] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.264448] LustreError: 8457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 319.464726] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.467954] LustreError: 6518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.484078] Lustre: Skipped 1 previous similar message [ 319.484922] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 319.484927] Lustre: Skipped 3 previous similar messages [ 319.521605] LustreError: 6518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 320.481119] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.501842] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 321.598996] Lustre: server umount lustre-MDT0001 complete [ 325.286877] Lustre: server umount lustre-OST0000 complete [ 328.959273] Lustre: server umount lustre-OST0001 complete [ 335.936663] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 345.897490] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 383.672666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.862471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 383.907987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 383.962199] Lustre: lustre-MDT0000: new disk, initializing [ 384.016270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.027818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 389.251036] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 399.830219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.937444] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 400.008058] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 400.012074] Lustre: Skipped 1 previous similar message [ 400.083934] Lustre: lustre-MDT0001: new disk, initializing [ 400.190699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 400.252126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 400.273094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 405.002676] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 415.770419] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 422.112256] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 429.222420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 429.457926] Lustre: lustre-OST0000: new disk, initializing [ 429.461828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 429.467199] Lustre: 17043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 429.603701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.888153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 430.915846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 431.215296] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 435.768455] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 445.791154] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 450.653820] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 450.896717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 455.747557] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 455.926904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 461.799690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 461.809205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.816552] LustreError: Skipped 1 previous similar message [ 461.826621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 461.835164] Lustre: Skipped 1 previous similar message [ 461.848536] Lustre: Skipped 4 previous similar messages [ 464.141619] Lustre: server umount lustre-OST0000 complete [ 472.039049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.046802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 472.048297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 472.052415] Lustre: Skipped 1 previous similar message [ 472.078748] Lustre: Skipped 1 previous similar message [ 474.344095] Lustre: server umount lustre-MDT0000 complete [ 477.156352] LustreError: 15083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.168983] LustreError: 15083:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 477.391654] LustreError: 15878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302542 with bad export cookie 3990272644285397545 [ 477.394197] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 477.416196] LustreError: 15878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 477.869446] Lustre: server umount lustre-MDT0001 complete [ 481.708888] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:29:05 (1789302545) [ 484.389329] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 485.974274] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:29:09 (1789302549) [ 493.437056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.993938] LustreError: 19451:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.227994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 499.331650] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 499.681168] LustreError: 19452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.691223] LustreError: 19452:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 509.083410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 514.084211] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 523.123658] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 530.033678] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 537.699723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 544.787339] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 552.634890] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 564.706227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.711497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 564.717536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 564.721832] Lustre: Skipped 1 previous similar message [ 564.981376] Lustre: server umount lustre-OST0000 complete [ 574.749983] Lustre: server umount lustre-MDT0000 complete [ 574.946356] LustreError: 20910:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.955493] LustreError: 20910:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 578.740477] LustreError: 21297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302643 with bad export cookie 3990272644285398420 [ 578.745190] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.747723] LustreError: 21297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 578.975823] Lustre: server umount lustre-MDT0001 complete [ 590.379265] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 594.368501] Key type lgssc unregistered [ 594.651180] LNet: 22984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 594.663497] LNetError: 22984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.675705] LNet: Removed LNI 192.168.202.111@tcp [ 595.839155] Key type .llcrypt unregistered [ 595.844243] Key type ._llcrypt unregistered [ 619.048318] Key type ._llcrypt registered [ 619.049954] Key type .llcrypt registered [ 619.131685] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 620.642016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 620.729470] alg: No test for adler32 (adler32-zlib) [ 621.763106] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 622.042055] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 623.760258] Key type lgssc registered [ 625.384691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 633.916930] loop: module loaded [ 634.012388] loop0: detected capacity change from 0 to 204800000 [ 634.329740] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 634.342559] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 634.661939] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 640.166772] loop0: detected capacity change from 0 to 204800000 [ 640.267730] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 640.275717] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 640.288773] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 647.151673] loop0: detected capacity change from 0 to 204800000 [ 654.086511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 654.111717] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.406490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 655.464610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 655.530940] Lustre: lustre-MDT0000: new disk, initializing [ 655.593811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 655.610939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 661.115145] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 670.841138] loop1: detected capacity change from 0 to 204800000 [ 677.050748] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 677.449516] Lustre: lustre-OST07c6: new disk, initializing [ 677.452283] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 677.457061] Lustre: Skipped 1 previous similar message [ 677.462049] Lustre: 26166:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 677.585676] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 683.732921] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 686.755942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 686.766386] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 686.821958] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 689.793469] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 689.801599] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 689.807461] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 689.812428] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 689.818967] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 689.823721] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 704.523328] Lustre: 26811:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 704.539260] Lustre: 26811:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 704.547657] Lustre: 26811:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 704.557605] Lustre: 26811:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 704.572287] Lustre: 26811:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 704.580764] Lustre: 26811:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 710.263909] Lustre: 26811:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 710.280748] Lustre: 26811:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 710.292816] Lustre: 26811:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 710.300558] Lustre: 26811:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.306514] Lustre: 26811:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 710.311635] Lustre: 26811:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.317043] Lustre: 26811:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 710.323720] Lustre: 26811:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.329422] Lustre: 26811:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 710.337620] Lustre: 26811:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.346703] Lustre: 26811:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 710.351381] Lustre: 26811:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.590727] Lustre: 26811:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 713.602036] Lustre: 26811:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 713.625894] Lustre: 26811:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 713.639040] Lustre: 26811:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.657200] Lustre: 26811:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 713.671541] Lustre: 26811:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.685164] Lustre: 26811:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 713.694033] Lustre: 26811:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.708491] Lustre: 26811:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 713.720949] Lustre: 26811:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.726495] Lustre: 26811:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 713.734702] Lustre: 26811:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 721.543923] Lustre: 25154:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 721.574226] Lustre: 25154:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 721.581557] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 721.594678] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 721.614508] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 721.622295] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 721.633098] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 721.649519] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 721.666263] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 721.688402] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 721.699406] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 721.712064] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 730.087716] Lustre: 26811:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 730.095840] Lustre: 26811:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 730.101676] Lustre: 26811:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 730.108644] Lustre: 26811:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 730.113238] Lustre: 26811:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 730.119652] Lustre: 26811:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 730.125616] Lustre: 26811:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 730.131220] Lustre: 26811:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 730.136596] Lustre: 26811:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 730.142160] Lustre: 26811:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 730.147357] Lustre: 26811:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 730.152841] Lustre: 26811:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 784.354123] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 784.355995] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 784.385904] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 788.185561] Lustre: server umount lustre-OST07c6 complete [ 798.057580] Lustre: server umount lustre-MDT0000 complete [ 813.484026] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 815.648411] Key type lgssc unregistered [ 815.943972] LNet: 28723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.953324] LNetError: 28723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.979835] LNet: Removed LNI 192.168.202.111@tcp [ 816.968329] Key type .llcrypt unregistered [ 816.974293] Key type ._llcrypt unregistered [ 834.769151] Key type ._llcrypt registered [ 834.775170] Key type .llcrypt registered [ 834.922758] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 835.830238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 835.945417] alg: No test for adler32 (adler32-zlib) [ 836.946211] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 837.126265] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 838.792753] Key type lgssc registered [ 839.535236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 846.260219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 846.273215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.523207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 852.656357] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 857.060081] LustreError: 29989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 857.098644] LustreError: 29989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 862.179622] LustreError: 29990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 863.712909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.973970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 869.275353] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 877.150835] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 883.501920] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 890.626913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 891.031063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.031556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 896.533786] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 903.242343] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 923.114051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 923.125394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.137524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 924.910493] Lustre: server umount lustre-OST0000 complete [ 930.785704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 930.791055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.801927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 930.814679] Lustre: Skipped 1 previous similar message [ 930.838321] Lustre: Skipped 1 previous similar message [ 935.380891] Lustre: server umount lustre-MDT0000 complete [ 935.907786] LustreError: 29985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.933771] LustreError: 29985:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 939.433368] LustreError: 29969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303004 with bad export cookie 12965869715517919805 [ 939.445059] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 939.449367] LustreError: 29969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 949.896500] Lustre: server umount lustre-MDT0001 complete [ 966.310312] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 970.302659] Key type lgssc unregistered [ 970.709618] LNet: 33727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.720813] LNetError: 33727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.743830] LNet: Removed LNI 192.168.202.111@tcp [ 971.503413] Key type .llcrypt unregistered [ 971.505088] Key type ._llcrypt unregistered [ 991.057534] Key type ._llcrypt registered [ 991.059916] Key type .llcrypt registered [ 991.210506] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 992.018980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 992.055077] alg: No test for adler32 (adler32-zlib) [ 993.114735] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 993.368449] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 995.048248] Key type lgssc registered [ 996.144967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1004.502704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1004.522399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.882722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.600056] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1016.302628] LustreError: 34995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1016.323517] LustreError: 34995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1019.741090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.191523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1024.871523] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1032.174713] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1038.603404] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1045.731268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1046.135753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.268279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1053.516165] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1061.395529] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1076.197271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1076.202666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.252508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1081.323906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1081.334265] Lustre: Skipped 2 previous similar messages [ 1081.550264] Lustre: server umount lustre-OST0000 complete [ 1086.433766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.451224] Lustre: Skipped 1 previous similar message [ 1086.460321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1086.465751] Lustre: Skipped 1 previous similar message [ 1091.554816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1091.563702] Lustre: Skipped 1 previous similar message [ 1092.333402] Lustre: server umount lustre-MDT0000 complete [ 1096.519787] LustreError: 34977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303161 with bad export cookie 2385856718110641975 [ 1096.553966] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1096.888333] Lustre: server umount lustre-MDT0001 complete [ 1109.239219] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 1112.054697] Key type lgssc unregistered [ 1112.346620] LNet: 38552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.355540] LNetError: 38552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.378652] LNet: Removed LNI 192.168.202.111@tcp [ 1113.259168] Key type .llcrypt unregistered [ 1113.261117] Key type ._llcrypt unregistered [ 1138.852733] Key type ._llcrypt registered [ 1138.853770] Key type .llcrypt registered [ 1138.955849] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 1140.111058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1140.366137] alg: No test for adler32 (adler32-zlib) [ 1141.726399] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 1142.091525] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 1143.760196] Key type lgssc registered [ 1145.330350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1153.105363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1153.124528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.429701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1158.526862] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1164.774405] LustreError: 39964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.810488] LustreError: 39964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1166.531233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1166.898990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1171.638480] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1179.612988] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1185.608753] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1192.443296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1192.782354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.519514] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1198.956500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1205.764130] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1218.023944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1218.047171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1218.083463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1219.074738] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.11@tcp (stopping) [ 1221.614232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1221.618641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1221.641281] Lustre: Skipped 1 previous similar message [ 1221.650075] Lustre: Skipped 2 previous similar messages [ 1223.599593] Lustre: server umount lustre-MDT0000 complete [ 1224.172140] LustreError: 39959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1226.723773] LustreError: 39965:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1226.739947] LustreError: 39965:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1228.575936] LustreError: 39945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303293 with bad export cookie 13342162135236230554 [ 1228.597253] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1229.275217] LustreError: 39961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1229.311506] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.11@tcp (stopping) [ 1231.843301] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.891647] LustreError: 40302:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.894308] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1233.924938] LustreError: 40302:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1233.950996] Lustre: Skipped 1 previous similar message [ 1234.998735] Lustre: server umount lustre-MDT0001 complete [ 1249.768608] Lustre: server umount lustre-OST0000 complete [ 1266.484481] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 1269.148480] Key type lgssc unregistered [ 1269.452721] LNet: 43821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.458050] LNetError: 43821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1270.507169] LNet: Removed LNI 192.168.202.111@tcp [ 1271.384770] Key type .llcrypt unregistered [ 1271.388805] Key type ._llcrypt unregistered [ 1292.585732] Key type ._llcrypt registered [ 1292.587081] Key type .llcrypt registered [ 1292.693996] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 1293.997987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1294.013678] alg: No test for adler32 (adler32-zlib) [ 1295.071241] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 1295.288294] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 1297.000253] Key type lgssc registered [ 1297.799201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1308.206826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1308.236690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1309.818562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1315.556487] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1319.400919] LustreError: 45234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.416321] LustreError: 45234:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1324.210926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1324.571215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1328.871627] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1337.443968] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1342.634132] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1349.161338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1349.475278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.609885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1355.349987] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1363.267440] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1374.183292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.187163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1374.194698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1374.211799] Lustre: Skipped 1 previous similar message [ 1375.394738] Lustre: server umount lustre-OST0000 complete [ 1392.635566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.647805] Lustre: Skipped 1 previous similar message [ 1392.652986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.723136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.730411] Lustre: Skipped 2 previous similar messages [ 1398.880194] Lustre: server umount lustre-MDT0000 complete [ 1399.781575] LustreError: 45233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.806522] LustreError: 45233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1402.140150] LustreError: 45212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303467 with bad export cookie 15157330051521688835 [ 1402.156034] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1402.160861] LustreError: 45212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1408.680087] Lustre: server umount lustre-MDT0001 complete [ 1424.036716] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 1426.446219] Key type lgssc unregistered [ 1426.759217] LNet: 48942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.765879] LNetError: 48942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1426.792479] LNet: Removed LNI 192.168.202.111@tcp [ 1428.100492] Key type .llcrypt unregistered [ 1428.102173] Key type ._llcrypt unregistered [ 1456.290356] Key type ._llcrypt registered [ 1456.296985] Key type .llcrypt registered [ 1456.460255] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 1457.537716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1457.576643] alg: No test for adler32 (adler32-zlib) [ 1458.839870] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 1459.175656] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 1460.882849] Key type lgssc registered [ 1462.335884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1470.887447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1470.898433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1472.236467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.425527] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1482.731831] LustreError: 50354:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.778000] LustreError: 50354:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1486.704126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1487.097481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1492.000555] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1501.719739] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1508.777354] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1516.556381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1516.804494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.828335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1523.479925] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1532.191451] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1539.611214] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1544.309605] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1556.716524] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-13 8h46m59s [ 1559.200269] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1563.112539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1563.120874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1563.140944] Lustre: Skipped 1 previous similar message [ 1563.164521] Lustre: Skipped 2 previous similar messages [ 1564.198829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1564.213711] Lustre: Skipped 2 previous similar messages [ 1565.676590] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.11@tcp (stopping) [ 1568.477433] Lustre: server umount lustre-MDT0000 complete [ 1570.276364] LustreError: 52753:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1570.313085] LustreError: 52753:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1572.248201] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1572.825963] LustreError: 50348:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1576.421123] LustreError: 50353:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1576.441581] LustreError: 50353:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1579.756361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.988321] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1580.446138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1580.542089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1581.506945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1585.305157] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1588.514944] Lustre: DEBUG MARKER: Wait for df (36753) ... [ 1590.123536] Lustre: DEBUG MARKER: done [ 1595.882951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1595.894232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.903041] Lustre: Skipped 1 previous similar message [ 1595.910183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1595.915334] Lustre: Skipped 1 previous similar message [ 1596.904393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1600.711497] Lustre: server umount lustre-OST0000 complete [ 1607.137510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1607.141614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1607.157462] Lustre: Skipped 1 previous similar message [ 1607.161353] Lustre: Skipped 2 previous similar messages [ 1611.230287] Lustre: server umount lustre-MDT0000 complete [ 1614.308552] LustreError: 50354:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.328192] LustreError: 50354:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1615.247798] LustreError: 50333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303680 with bad export cookie 15857193002279535498 [ 1615.256049] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1615.257306] LustreError: 50333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1615.526511] Lustre: server umount lustre-MDT0001 complete [ 1629.732772] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 1632.546221] Key type lgssc unregistered [ 1632.822419] LNet: 55279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1632.834513] LNetError: 55279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1632.858869] LNet: Removed LNI 192.168.202.111@tcp [ 1633.971990] Key type .llcrypt unregistered [ 1633.977455] Key type ._llcrypt unregistered [ 1677.860251] Key type ._llcrypt registered [ 1677.862967] Key type .llcrypt registered [ 1677.941163] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 1678.950704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1678.987517] alg: No test for adler32 (adler32-zlib) [ 1680.116869] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 1680.336068] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 1682.024263] Key type lgssc registered [ 1683.097882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1691.222245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1691.241054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.538813] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1692.581499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1692.718265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.217548] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1705.817966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.894623] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1705.915838] Lustre: 57482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1706.090317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1710.415205] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1718.810679] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1723.968685] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1732.065050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1732.440496] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1732.805930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.976792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1739.610579] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1749.173424] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1756.951546] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-13 8h50m19s [ 1758.745862] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1768.984872] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1772.429681] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1772.809280] LustreError: 58924:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1793.009336] Lustre: lustre-MDT0000: Client 5f0cb03a-df03-446b-ae0b-0148a09074ad (at 192.168.202.11@tcp) reconnecting [ 1812.905136] LustreError: 58924:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1814.775711] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1817.112504] Lustre: DEBUG MARKER: done [ 1826.334841] Lustre: server umount lustre-OST0000 complete [ 1831.396973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1831.430461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1831.437136] Lustre: Skipped 1 previous similar message [ 1831.458665] Lustre: Skipped 1 previous similar message [ 1832.481552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1832.493855] Lustre: Skipped 1 previous similar message [ 1838.567119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1838.574110] Lustre: Skipped 1 previous similar message [ 1842.656163] Lustre: 56807:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789303891/real 1789303891] req@0000000050ad831f x1876220963774848/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789303907 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1842.659964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.690964] Lustre: 56807:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1843.748825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1845.217096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1845.410942] Lustre: server umount lustre-MDT0000 complete [ 1847.842262] Lustre: 56805:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789303896/real 1789303896] req@000000003d04bfbb x1876220963777024/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789303912 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1848.524533] LustreError: 57475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303913 with bad export cookie 2408124777905102347 [ 1848.534988] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1848.706193] Lustre: server umount lustre-MDT0001 complete [ 1860.597213] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 1863.539522] Key type lgssc unregistered [ 1863.838800] LNet: 61815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1863.850417] LNetError: 61815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1863.870612] LNet: Removed LNI 192.168.202.111@tcp [ 1864.652384] Key type .llcrypt unregistered [ 1864.653909] Key type ._llcrypt unregistered [ 1909.951955] Key type ._llcrypt registered [ 1909.953450] Key type .llcrypt registered [ 1910.068253] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 1911.644973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1911.675733] alg: No test for adler32 (adler32-zlib) [ 1912.787939] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 1913.055944] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 1914.792339] Key type lgssc registered [ 1915.922823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1919.247624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1919.277493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1920.595773] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1920.638934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1920.801293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.774867] Lustre: Failing over lustre-MDT0000 [ 1922.260099] Lustre: server umount lustre-MDT0000 complete [ 1928.509737] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:53:11 (1789303991) [ 1934.874626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.299352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.312864] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1947.961852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.105649] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1948.130687] Lustre: 64612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1948.440462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1953.346049] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1961.581986] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1968.393542] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1976.923382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1977.241418] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1977.466717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.279898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1986.636493] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1996.105912] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2001.663153] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2008.036228] Lustre: server umount lustre-MDT0000 complete [ 2010.092060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2010.108422] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.131834] Lustre: Skipped 2 previous similar messages [ 2012.779168] LustreError: 66468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304077 with bad export cookie 8423582585010973601 [ 2012.781401] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2012.792939] LustreError: 66468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2013.148100] Lustre: server umount lustre-MDT0001 complete [ 2017.537484] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2024.179847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2031.585846] Lustre: 63537:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789304080/real 1789304080] req@00000000a7cd56b7 x1876221208075648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789304096 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2031.623903] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2033.653780] Lustre: 63536:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789304082/real 1789304082] req@000000004fea3c61 x1876221208075776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789304098 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2035.139874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.704508] Lustre: 63534:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789304085/real 1789304085] req@000000000891e492 x1876221208075904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789304101 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2037.473783] LustreError: 68368:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2037.478412] LustreError: 68368:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000fc512bb5 x1876221208076032/t0(0) o250->MGC192.168.202.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789304102 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2037.496559] LustreError: 68368:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2037.795331] LustreError: 63533:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000029854424 x1876221208076800/t0(0) o250->MGC192.168.202.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2038.186584] LustreError: 66462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2038.204200] LustreError: 66462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2038.290753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.454136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2039.264487] Lustre: 63534:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789304087/real 1789304087] req@000000009f791402 x1876221208076160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789304103 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2041.382497] LustreError: 68380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.393089] LustreError: 68380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2043.406237] Lustre: 63536:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789304092/real 1789304092] req@000000001b67f827 x1876221208076288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789304108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2043.540703] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2046.440033] LustreError: 66462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.458106] LustreError: 66462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2050.548893] LustreError: 68983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.559538] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2050.575116] LustreError: 68983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2052.826912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.193971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2058.230690] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2058.874055] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2067.047830] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2074.059518] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2083.824362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2083.828295] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.842382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2083.851921] Lustre: Skipped 1 previous similar message [ 2083.862623] Lustre: Skipped 2 previous similar messages [ 2088.932093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2088.943180] Lustre: Skipped 2 previous similar messages [ 2090.256449] Lustre: server umount lustre-MDT0000 complete [ 2094.067706] LustreError: 68382:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2094.096326] LustreError: 68382:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2095.037952] LustreError: 66468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304160 with bad export cookie 8423582585010975876 [ 2095.039361] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2095.052244] LustreError: 66468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2095.524195] Lustre: server umount lustre-MDT0001 complete [ 2103.006911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2112.349742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.552695] Lustre: 63536:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789304164/real 1789304164] req@000000006ce28238 x1876221208104832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789304180 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2115.590305] Lustre: 63536:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2115.603922] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.631439] Lustre: Skipped 1 previous similar message [ 2121.145361] LustreError: 69690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.221731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.279396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2124.264337] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2125.640641] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2133.030423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.974324] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2138.609688] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2146.378904] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2154.384449] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2164.212364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.220554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.241255] Lustre: Skipped 2 previous similar messages [ 2164.262539] Lustre: Skipped 2 previous similar messages [ 2168.432906] Lustre: server umount lustre-MDT0000 complete [ 2169.315508] LustreError: 72313:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.354429] LustreError: 72313:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2173.035664] LustreError: 66469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304238 with bad export cookie 8423582585010977255 [ 2173.049607] LustreError: 66469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2173.053104] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2173.389940] Lustre: server umount lustre-MDT0001 complete [ 2179.643779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2183.173994] Lustre: DEBUG MARKER: files compared the same [ 2195.928517] Lustre: server umount lustre-OST0000 complete [ 2213.967871] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 2216.888749] Key type lgssc unregistered [ 2217.242679] LNet: 74343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.255040] LNetError: 74343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.285743] LNet: Removed LNI 192.168.202.111@tcp [ 2218.435710] Key type .llcrypt unregistered [ 2218.438158] Key type ._llcrypt unregistered [ 2243.335495] Key type ._llcrypt registered [ 2243.337873] Key type .llcrypt registered [ 2243.453647] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 2244.304531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2244.533869] alg: No test for adler32 (adler32-zlib) [ 2245.707679] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 2246.000761] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 2247.728964] Key type lgssc registered [ 2249.108977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2258.075722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2258.090686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.363606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.081307] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug +malloc all [ 2269.666554] LustreError: 75759:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.675023] LustreError: 75759:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2274.789113] LustreError: 75760:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.178446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.455969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2282.539079] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug +malloc all [ 2290.730539] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2297.304906] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2304.270135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2304.575812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.759919] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug +malloc all [ 2320.223852] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2329.192641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2339.809648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2339.814180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2339.829479] Lustre: Skipped 1 previous similar message [ 2344.683468] Lustre: server umount lustre-OST0000 complete [ 2349.025863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2349.031366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2349.039286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2349.050226] Lustre: Skipped 1 previous similar message [ 2350.052071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2350.053264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.169090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2355.172350] Lustre: Skipped 2 previous similar messages [ 2360.303422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2362.336161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2362.459961] Lustre: server umount lustre-MDT0000 complete [ 2365.412675] LustreError: 76487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.433210] LustreError: 76487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2366.762154] LustreError: 75740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304431 with bad export cookie 6562404431418469614 [ 2366.789425] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2367.346335] Lustre: server umount lustre-MDT0001 complete [ 2379.970138] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 2383.156864] Key type lgssc unregistered [ 2383.601783] LNet: 79313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2383.630227] LNetError: 79313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2383.654524] LNet: Removed LNI 192.168.202.111@tcp [ 2384.496372] Key type .llcrypt unregistered [ 2384.498772] Key type ._llcrypt unregistered [ 2408.219589] Key type ._llcrypt registered [ 2408.226498] Key type .llcrypt registered [ 2408.336553] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 2409.889363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2409.956937] alg: No test for adler32 (adler32-zlib) [ 2410.996289] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 2411.237907] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 2412.912381] Key type lgssc registered [ 2413.995739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2422.275320] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2422.295235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2437.856424] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2442.976332] LustreError: 80715:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.111@tcp: failed processing log, type 4: rc = -110 [ 2469.665761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2477.097682] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2485.910197] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2494.438239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2494.556382] Lustre: *** cfs_fail_loc=706, val=0*** [ 2495.023166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2495.094151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2499.630453] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2500.067644] LustreError: 80741:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2500.086426] LustreError: 80741:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2504.171191] LustreError: 80740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.188181] LustreError: 80740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2508.602821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.144937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2513.855128] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2521.459438] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2527.306441] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2529.780203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2529.783071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.791502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2529.819556] Lustre: Skipped 1 previous similar message [ 2534.884158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2534.900116] Lustre: Skipped 2 previous similar messages [ 2535.491893] Lustre: server umount lustre-OST0000 complete [ 2540.010857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2540.012083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.043329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2540.053314] Lustre: Skipped 1 previous similar message [ 2545.124167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2545.134531] Lustre: Skipped 1 previous similar message [ 2546.371790] Lustre: server umount lustre-MDT0000 complete [ 2550.244161] LustreError: 81804:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.265872] LustreError: 81804:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2550.440543] LustreError: 80721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304615 with bad export cookie 304238108990003580 [ 2550.444714] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2550.457347] LustreError: 80721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2550.896095] Lustre: server umount lustre-MDT0001 complete [ 2563.464134] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 2566.350254] Key type lgssc unregistered [ 2566.628558] LNet: 84330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2566.632471] LNetError: 84330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2566.648240] LNet: Removed LNI 192.168.202.111@tcp [ 2567.614806] Key type .llcrypt unregistered [ 2567.617451] Key type ._llcrypt unregistered [ 2591.636447] Key type ._llcrypt registered [ 2591.644719] Key type .llcrypt registered [ 2591.774926] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 2593.283773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2593.318906] alg: No test for adler32 (adler32-zlib) [ 2594.449861] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 2594.898133] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 2596.592225] Key type lgssc registered [ 2597.763592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2606.400923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2606.424644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2612.627988] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2620.187376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2620.619499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2624.573920] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2626.018775] LustreError: 86437:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.032125] LustreError: 86437:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2630.118234] LustreError: 86441:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.147045] LustreError: 86441:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2632.529557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2632.886492] LustreError: 86437:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.931205] LustreError: 86437:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2632.949452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2638.307556] LustreError: 86440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.323676] LustreError: 86440:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2638.500558] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2643.429949] LustreError: 87183:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.460055] LustreError: 87183:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2647.918523] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2647.932909] Lustre: MGS: Client 7ed318f4-d954-4d53-9769-3b4b92494ba3 (at 0@lo) reconnecting [ 2647.957295] LustreError: 86452:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2648.206613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2648.312122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2653.603493] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2669.033229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2669.038342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.050523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2669.058720] Lustre: Skipped 1 previous similar message [ 2673.079713] Lustre: server umount lustre-OST0000 complete [ 2679.269386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.291642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.300785] Lustre: Skipped 1 previous similar message [ 2683.484026] Lustre: server umount lustre-MDT0000 complete [ 2684.393752] LustreError: 86441:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.412977] LustreError: 86441:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2687.584791] Lustre: server umount lustre-MDT0001 complete [ 2691.657819] Lustre: server umount MGS complete [ 2702.735615] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:06:06 (1789304766) [ 2734.041511] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 2744.910665] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 2785.916550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2790.750581] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2801.374124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2801.701145] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2801.757855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2801.845702] Lustre: lustre-MDT0001: new disk, initializing [ 2801.925632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2801.964332] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2801.975213] Lustre: 93607:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2801.986918] LustreError: 93607:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2806.743499] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2817.814643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2818.141739] Lustre: lustre-OST0000: new disk, initializing [ 2818.144740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2818.149962] Lustre: 94372:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2818.264994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.604900] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2828.388664] Lustre: 94803:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000812c9fb4 x1876221923205120/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789304909 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2836.267801] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2843.495788] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2843.514992] Lustre: MGS: Client 0bdf11b8-d14d-4ebc-abf1-237cc5181416 (at 0@lo) reconnecting [ 2843.552891] LustreError: 93593:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2843.586599] Lustre: 92874:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2843.620447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2843.711506] Lustre: lustre-MDT0000: new disk, initializing [ 2843.863327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2843.887764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2843.907257] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2844.960993] Lustre: 95442:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008f5d0e92 x1876221923212032/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789304926 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2848.270096] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2850.036424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2850.050628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2850.137975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2875.367059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2875.383988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2875.405204] Lustre: Skipped 2 previous similar messages [ 2875.427832] Lustre: Skipped 1 previous similar message [ 2879.542846] Lustre: server umount lustre-OST0000 complete [ 2885.606550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.611296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2885.616497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2885.616505] LustreError: Skipped 1 previous similar message [ 2885.622043] Lustre: Skipped 1 previous similar message [ 2885.646482] Lustre: Skipped 1 previous similar message [ 2889.692964] Lustre: server umount lustre-MDT0000 complete [ 2890.725423] LustreError: 93578:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.755282] LustreError: 93578:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2900.302776] Lustre: server umount lustre-MDT0001 complete [ 2904.686616] Lustre: server umount MGS complete [ 2916.198913] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:09:39 (1789304979) [ 2935.392779] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 2938.067917] Key type lgssc unregistered [ 2938.297772] LNet: 97938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2938.307583] LNetError: 97938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2938.332697] LNet: Removed LNI 192.168.202.111@tcp [ 2939.093709] Key type .llcrypt unregistered [ 2939.095966] Key type ._llcrypt unregistered [ 2956.904368] Key type ._llcrypt registered [ 2956.908454] Key type .llcrypt registered [ 2957.022921] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 2958.065746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2958.094451] alg: No test for adler32 (adler32-zlib) [ 2959.316807] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 2959.620774] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 2961.304831] Key type lgssc registered [ 2962.344602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2970.425472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2970.442269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2971.977075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2976.990784] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2982.377931] LustreError: 99156:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2982.390279] LustreError: 99156:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2985.958747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.344914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2991.693159] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2998.730609] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3005.047497] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3011.672584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3012.027738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3017.176364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3018.856519] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3027.066905] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3041.765248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.771470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3041.794720] Lustre: Skipped 1 previous similar message [ 3041.813780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3046.886591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3046.901553] Lustre: Skipped 2 previous similar messages [ 3052.000151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3052.012995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3052.023700] Lustre: Skipped 1 previous similar message [ 3052.176025] Lustre: server umount lustre-OST0000 complete [ 3057.124529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3057.136433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3057.143384] Lustre: Skipped 1 previous similar message [ 3061.106471] Lustre: server umount lustre-MDT0000 complete [ 3062.244232] LustreError: 99149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.265346] LustreError: 99149:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3064.211970] LustreError: 99135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305129 with bad export cookie 2942897606560673019 [ 3064.227583] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3064.575643] Lustre: server umount lustre-MDT0001 complete [ 3074.976415] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 3077.634457] Key type lgssc unregistered [ 3077.897899] LNet: 102687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3077.911668] LNetError: 102687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3077.943868] LNet: Removed LNI 192.168.202.111@tcp [ 3078.919296] Key type .llcrypt unregistered [ 3078.923088] Key type ._llcrypt unregistered [ 3095.798269] Key type ._llcrypt registered [ 3095.800467] Key type .llcrypt registered [ 3095.914277] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 3097.182527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3097.194809] alg: No test for adler32 (adler32-zlib) [ 3098.303757] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 3098.631611] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 3100.290902] Key type lgssc registered [ 3101.593803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3107.730116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3107.732476] LustreError: 103768:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3107.738241] Lustre: Skipped 1 previous similar message [ 3107.746764] LustreError: 103767:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3107.750506] LustreError: 103768:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 3107.755204] LustreError: 103768:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3107.769054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.770563] LustreError: 103768:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3109.279187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.821285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.042952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3121.313488] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3126.591188] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing lsmod [ 3129.079396] LustreError: 104971:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3129.092420] LustreError: 104974:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3129.097033] LustreError: 104971:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 3129.104167] LustreError: 104971:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3129.112353] LustreError: 104971:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3129.113244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3129.454104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3132.393641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3132.404306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3132.425438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3134.947063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3134.953926] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.962611] Lustre: Skipped 1 previous similar message [ 3138.284748] Lustre: server umount lustre-MDT0000 complete [ 3140.065930] LustreError: 104349:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.079925] LustreError: 104349:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3141.624208] LustreError: 103785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305206 with bad export cookie 2763561616776149550 [ 3141.637705] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3141.643059] LustreError: 103785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3141.949417] Lustre: server umount lustre-MDT0001 complete [ 3155.526100] Lustre: server umount lustre-OST0000 complete [ 3162.523249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3162.877765] LustreError: 106036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3162.940173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3166.097945] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3168.225798] LustreError: 106037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3172.780659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.045221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3177.215705] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3184.481883] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3189.891971] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3196.376811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3196.581304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3202.696943] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3208.878479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3209.652359] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3219.427140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3219.432786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.436992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3219.437016] Lustre: Skipped 2 previous similar messages [ 3219.463966] Lustre: Skipped 1 previous similar message [ 3221.196961] Lustre: server umount lustre-OST0000 complete [ 3229.668043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3229.670200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3229.670552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3229.670557] Lustre: Skipped 1 previous similar message [ 3229.680202] Lustre: Skipped 1 previous similar message [ 3229.687491] LustreError: Skipped 1 previous similar message [ 3230.918282] Lustre: server umount lustre-MDT0000 complete [ 3234.791153] LustreError: 106036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.815367] LustreError: 106036:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3234.892746] LustreError: 106016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305300 with bad export cookie 2763561616776150229 [ 3234.907583] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3235.332533] Lustre: server umount lustre-MDT0001 complete [ 3244.738746] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 3247.322224] Key type lgssc unregistered [ 3247.570738] LNet: 109566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3247.589750] LNetError: 109566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3248.615228] LNet: Removed LNI 192.168.202.111@tcp [ 3249.520444] Key type .llcrypt unregistered [ 3249.523453] Key type ._llcrypt unregistered [ 3268.637813] Key type ._llcrypt registered [ 3268.640965] Key type .llcrypt registered [ 3268.743085] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 3270.012867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3270.069741] alg: No test for adler32 (adler32-zlib) [ 3271.126841] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 3271.341272] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 3273.058356] Key type lgssc registered [ 3273.996987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3280.697562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3280.709444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.116983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3285.976473] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3292.642680] LustreError: 110961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3292.657828] LustreError: 110961:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3293.536621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.863649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3298.366237] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3304.786948] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3310.117369] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3316.350859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3316.671661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3322.460898] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3326.909270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3328.899755] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3333.208958] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3336.449628] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3341.746838] Lustre: server umount lustre-OST0000 complete [ 3345.378745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3345.382984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.392883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3347.437491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3347.443156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.562107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3352.567980] Lustre: Skipped 2 previous similar messages [ 3355.690194] Lustre: server umount lustre-MDT0000 complete [ 3357.671038] LustreError: 112000:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.687681] LustreError: 112000:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3358.689592] Lustre: 110272:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789305407/real 1789305407] req@0000000094883298 x1876222632038272/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789305423 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3358.722173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3358.742543] LustreError: 110961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3358.763045] LustreError: 110961:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3359.302612] LustreError: 110941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305424 with bad export cookie 12576080669911563572 [ 3359.312295] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3359.610389] Lustre: server umount lustre-MDT0001 complete [ 3369.605963] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 3372.377124] Key type lgssc unregistered [ 3372.620701] LNet: 114604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3372.625690] LNetError: 114604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3372.645854] LNet: Removed LNI 192.168.202.111@tcp [ 3373.419775] Key type .llcrypt unregistered [ 3373.421393] Key type ._llcrypt unregistered [ 3387.558988] Key type ._llcrypt registered [ 3387.562644] Key type .llcrypt registered [ 3387.650456] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 3388.546145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3388.673047] alg: No test for adler32 (adler32-zlib) [ 3389.703412] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 3389.925792] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 3391.578754] Key type lgssc registered [ 3392.570588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3398.450354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3398.460627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.789911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.547824] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3409.400313] LustreError: 115802:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3409.418811] LustreError: 115802:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3411.552099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3411.924508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3416.260661] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3423.913197] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3429.442366] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3435.357353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3435.699630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.833047] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3443.893503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3448.334954] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3458.028480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3458.037748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3458.058446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3458.530334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3458.539102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3458.560369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3462.785935] Lustre: server umount lustre-OST0000 complete [ 3468.257684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3468.262915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.273779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3468.281178] Lustre: Skipped 2 previous similar messages [ 3472.552334] Lustre: server umount lustre-MDT0000 complete [ 3474.915861] LustreError: 115798:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3474.927076] LustreError: 115798:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3475.665874] LustreError: 115783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305540 with bad export cookie 6243861735556038344 [ 3475.675026] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3475.689453] LustreError: 115783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3475.930850] Lustre: server umount lustre-MDT0001 complete [ 3486.045905] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 3488.223521] Key type lgssc unregistered [ 3488.492896] LNet: 119330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3488.499446] LNetError: 119330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3489.510563] LNet: Removed LNI 192.168.202.111@tcp [ 3490.293877] Key type .llcrypt unregistered [ 3490.297535] Key type ._llcrypt unregistered [ 3508.953799] Key type ._llcrypt registered [ 3508.956582] Key type .llcrypt registered [ 3509.045510] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 3509.937871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3509.989892] alg: No test for adler32 (adler32-zlib) [ 3511.021395] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 3511.267682] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 3512.920201] Key type lgssc registered [ 3513.899188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3520.927362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3520.940110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3522.334914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.378747] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3532.781115] LustreError: 120720:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3532.796989] LustreError: 120720:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3535.023983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3535.401161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3540.263705] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3546.676842] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3551.199659] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3556.572685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3556.886479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3562.555871] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3567.217279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3569.147558] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3576.015780] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3578.977240] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3578.981911] Lustre: Skipped 2 previous similar messages [ 3582.049308] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3582.101726] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3582.658710] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3582.664770] Lustre: Skipped 2 previous similar messages [ 3589.205452] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3589.211581] Lustre: Skipped 1 previous similar message [ 3594.365082] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.11@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3594.373502] Lustre: Skipped 2 previous similar messages [ 3595.361765] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.11@tcp 192.168.0.[2,10]@tcp [ 3603.969676] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.11/24@tcp in log lustre-MDT0000 [ 3603.975305] Lustre: Skipped 2 previous similar messages [ 3613.230932] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.11/24@tcp [ 3613.241179] Lustre: Skipped 1 previous similar message [ 3618.785455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3618.789307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3618.794996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3618.811703] Lustre: Skipped 1 previous similar message [ 3623.402764] Lustre: server umount lustre-OST0000 complete [ 3627.489907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3627.510277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3627.518953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3627.521485] Lustre: Skipped 1 previous similar message [ 3629.026766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3629.027812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3632.873433] Lustre: server umount lustre-MDT0000 complete [ 3634.145280] LustreError: 121719:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3634.173882] LustreError: 121719:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3636.776480] LustreError: 124413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305701 with bad export cookie 13223622223288367346 [ 3636.780366] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3636.798047] LustreError: 124413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3637.135664] Lustre: server umount lustre-MDT0001 complete [ 3647.076767] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 3649.390648] Key type lgssc unregistered [ 3649.661755] LNet: 125763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3649.666583] LNetError: 125763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3649.685655] LNet: Removed LNI 192.168.202.111@tcp [ 3650.470202] Key type .llcrypt unregistered [ 3650.472302] Key type ._llcrypt unregistered [ 3673.990642] Key type ._llcrypt registered [ 3673.995864] Key type .llcrypt registered [ 3674.089875] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 3675.089392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3675.120641] alg: No test for adler32 (adler32-zlib) [ 3676.229271] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 3676.449102] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 3678.104276] Key type lgssc registered [ 3678.875738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3684.843752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3684.865538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3686.396926] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3686.427961] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.11/24@tcp [ 3686.428246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3689.901826] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3696.611149] LustreError: 127299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3696.619541] LustreError: 127299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3697.156144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3697.430799] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3697.452751] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.11/24@tcp [ 3697.539190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3701.469733] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3708.098548] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3713.646471] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3719.631891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3719.890249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3726.125327] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3731.217297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3731.756680] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3741.670809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3741.672034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3741.687375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3741.693767] Lustre: Skipped 1 previous similar message [ 3746.791366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3746.796423] Lustre: Skipped 1 previous similar message [ 3747.161502] Lustre: server umount lustre-OST0000 complete [ 3751.908633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3751.917223] Lustre: Skipped 2 previous similar messages [ 3751.920626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3756.872942] Lustre: server umount lustre-MDT0000 complete [ 3757.029749] LustreError: 128359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3757.044183] LustreError: 128359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3759.869031] LustreError: 127279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305825 with bad export cookie 475370124759754509 [ 3759.871538] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3759.883356] LustreError: 127279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3760.169252] Lustre: server umount lustre-MDT0001 complete [ 3768.951642] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 3771.083668] Key type lgssc unregistered [ 3771.312619] LNet: 130885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3771.315828] LNetError: 130885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3771.325041] LNet: Removed LNI 192.168.202.111@tcp [ 3772.051697] Key type .llcrypt unregistered [ 3772.053474] Key type ._llcrypt unregistered [ 3792.589351] Key type ._llcrypt registered [ 3792.590973] Key type .llcrypt registered [ 3792.671919] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 3806.004401] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 3806.905393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3806.926988] alg: No test for adler32 (adler32-zlib) [ 3808.024297] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 3808.204469] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 3809.832749] Key type lgssc registered [ 3811.004711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3839.535940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3839.572128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3840.787948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3840.813098] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3840.868484] Lustre: lustre-MDT0000: new disk, initializing [ 3840.927689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3840.941802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3844.894349] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3853.957400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3854.078430] Lustre: 134278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3854.120866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3854.124452] Lustre: Skipped 1 previous similar message [ 3854.182668] Lustre: lustre-MDT0001: new disk, initializing [ 3854.253330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3854.280711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3854.290184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3857.404218] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3864.537281] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3869.179692] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3874.710582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3874.973374] Lustre: lustre-OST0000: new disk, initializing [ 3874.976841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3874.983988] Lustre: 136243:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3875.074693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3880.071992] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3882.052535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3882.060281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3882.156632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3887.746847] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3891.388217] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3891.511984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3895.187810] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3895.351942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3897.829719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3897.831572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3897.835721] LustreError: Skipped 1 previous similar message [ 3897.837155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3897.850849] Lustre: Skipped 1 previous similar message [ 3902.953814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3902.958409] Lustre: Skipped 2 previous similar messages [ 3903.229748] Lustre: server umount lustre-OST0000 complete [ 3908.069426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3908.071053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3908.079228] Lustre: Skipped 1 previous similar message [ 3913.185071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3913.189956] Lustre: Skipped 2 previous similar messages [ 3918.305758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3918.308655] Lustre: Skipped 1 previous similar message [ 3920.352118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3920.473351] Lustre: server umount lustre-MDT0000 complete [ 3922.915181] LustreError: 134270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305988 with bad export cookie 3708584316009174569 [ 3922.917583] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3922.923535] LustreError: 134270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3923.147190] Lustre: server umount lustre-MDT0001 complete [ 3929.304117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3929.725790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3933.150677] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3935.202223] LustreError: 138461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3935.213282] LustreError: 138461:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3939.267752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3939.303757] LustreError: 138462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3939.602960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3942.663315] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3948.144672] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3952.536808] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3956.939725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3957.120134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3961.332860] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3966.871733] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3970.578989] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3974.927843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3978.152612] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3978.314349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3985.904461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3986.063545] Lustre: lustre-OST0001: new disk, initializing [ 3986.067552] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3986.074139] Lustre: 141860:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3986.149697] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3990.952985] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3992.628373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3992.636949] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3992.728671] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3998.030405] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3998.199725] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4001.633144] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4001.787790] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4006.161124] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9683888a9000.ost_server_uuid 50 [ 4007.596961] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9683888a9000.ost_server_uuid in FULL state after 0 sec [ 4018.662897] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4018.663730] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.669807] LustreError: Skipped 1 previous similar message [ 4018.672825] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4018.684227] Lustre: Skipped 1 previous similar message [ 4020.449760] Lustre: server umount lustre-OST0001 complete [ 4023.781548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4023.786824] LustreError: 142325:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4023.803344] Lustre: Skipped 1 previous similar message [ 4023.840445] LustreError: 142325:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4028.903217] LustreError: 140282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4028.910109] LustreError: 140282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4029.443783] Lustre: server umount lustre-OST0000 complete [ 4034.020139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4034.026874] Lustre: Skipped 1 previous similar message [ 4038.491526] Lustre: server umount lustre-MDT0000 complete [ 4039.147046] LustreError: 138457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4039.164415] LustreError: 138457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4041.128488] LustreError: 138441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306106 with bad export cookie 3708584316009175451 [ 4041.132925] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4041.135850] LustreError: 138441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4041.425600] Lustre: server umount lustre-MDT0001 complete [ 4054.459288] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 4056.271918] Key type lgssc unregistered [ 4056.521718] LNet: 144858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4056.525266] LNetError: 144858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4056.533177] LNet: Removed LNI 192.168.202.111@tcp [ 4057.191632] Key type .llcrypt unregistered [ 4057.193719] Key type ._llcrypt unregistered [ 4085.134727] Key type ._llcrypt registered [ 4085.137111] Key type .llcrypt registered [ 4085.200834] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 4095.867773] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 4096.647133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4096.666842] alg: No test for adler32 (adler32-zlib) [ 4097.610848] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 4097.780024] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 4099.416202] Key type lgssc registered [ 4100.242793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4121.808446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4121.822739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4122.984240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4123.005660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4123.057356] Lustre: lustre-MDT0000: new disk, initializing [ 4123.098828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4123.115303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4125.784837] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4132.928768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.053425] Lustre: 148984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4133.112209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4133.115828] Lustre: Skipped 1 previous similar message [ 4133.177878] Lustre: lustre-MDT0001: new disk, initializing [ 4133.266300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4133.297595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4133.308577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4136.223278] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4142.302398] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4145.853226] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4150.294339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4150.489959] Lustre: lustre-OST0000: new disk, initializing [ 4150.492674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4150.498221] Lustre: 150949:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4150.586165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.120870] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4156.970210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4156.974271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4157.030431] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4161.701159] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4165.083027] Lustre: Failing over lustre-OST0000 [ 4165.211574] Lustre: server umount lustre-OST0000 complete [ 4181.846787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4182.121246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4182.145209] Lustre: Skipped 1 previous similar message [ 4182.166915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.181821] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4183.137383] Lustre: 147208:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789306232/real 1789306232] req@0000000087623add x1876223498156416/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789306248 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4184.041143] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4186.438628] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4187.664560] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4187.667386] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4188.704413] Lustre: 147210:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789306237/real 1789306237] req@00000000570047dc x1876223498157568/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789306253 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4188.720855] Lustre: 147210:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4189.041257] Lustre: Failing over lustre-MDT0000 [ 4189.165223] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.11@tcp (stopping) [ 4189.666851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4189.675794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4191.253599] Lustre: server umount lustre-MDT0000 complete [ 4192.742673] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4193.763347] Lustre: 147210:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789306242/real 1789306242] req@000000003d8f7509 x1876223498158720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789306258 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4197.858032] LustreError: 150149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4197.872241] LustreError: 150149:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 4199.384416] LustreError: 148990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4202.985872] LustreError: 151871:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4202.998922] LustreError: 151871:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4206.086089] LustreError: 148990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4206.102336] LustreError: 148990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4207.460260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.571572] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4207.761985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4207.800194] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4210.720484] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4211.165943] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4213.233547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4213.239353] Lustre: Skipped 1 previous similar message [ 4213.260481] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4213.308426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4218.343193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4218.345744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.360141] Lustre: Skipped 1 previous similar message [ 4218.367869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4218.372810] Lustre: Skipped 1 previous similar message [ 4222.719883] Lustre: server umount lustre-OST0000 complete [ 4228.578481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4228.579870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.581450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4228.598427] Lustre: Skipped 2 previous similar messages [ 4231.476831] Lustre: server umount lustre-MDT0000 complete [ 4233.704378] LustreError: 148991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.716862] LustreError: 148991:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4234.258193] LustreError: 148977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306299 with bad export cookie 10165188602707145904 [ 4234.269733] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4234.271831] LustreError: 148977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4234.536470] Lustre: server umount lustre-MDT0001 complete [ 4242.352641] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 4244.200330] Key type lgssc unregistered [ 4244.438889] LNet: 154604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4244.444935] LNetError: 154604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4244.455828] LNet: Removed LNI 192.168.202.111@tcp [ 4245.070866] Key type .llcrypt unregistered [ 4245.072394] Key type ._llcrypt unregistered [ 4277.297572] Key type ._llcrypt registered [ 4277.299606] Key type .llcrypt registered [ 4277.409412] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 4278.478275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4278.578136] alg: No test for adler32 (adler32-zlib) [ 4279.605876] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 4279.861740] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 4281.552231] Key type lgssc registered [ 4282.346538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4288.091247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4288.103585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4289.368648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4292.577668] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4299.745918] LustreError: 156136:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4299.756837] LustreError: 156136:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4300.446422] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4303.672406] LustreError: 156165:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4303.679160] LustreError: 156165:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4303.686948] LustreError: 156165:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 4303.923939] Lustre: server umount lustre-MDT0000 complete [ 4314.461423] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 4321.341772] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 4346.760037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4346.930344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4346.948057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4346.998855] Lustre: lustre-MDT0000: new disk, initializing [ 4347.032191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4347.074079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4349.835399] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4357.540391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4357.628886] Lustre: 159466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4357.670305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4357.674903] Lustre: Skipped 1 previous similar message [ 4357.722302] Lustre: lustre-MDT0001: new disk, initializing [ 4357.790319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4357.824513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4357.834820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4360.952335] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4367.804882] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4371.787578] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4376.656519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4376.816709] Lustre: lustre-OST0000: new disk, initializing [ 4376.819548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4376.827718] Lustre: 161434:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4376.895028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4381.073823] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4383.311987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4383.317622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4383.371218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4387.918281] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4391.974457] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4393.206273] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:34:16 (1789306456) [ 4573.152454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4573.155572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4573.160467] Lustre: Skipped 1 previous similar message [ 4573.162915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4577.191279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4577.199060] Lustre: Skipped 4 previous similar messages [ 4579.423451] Lustre: server umount lustre-MDT0000 complete [ 4581.866175] LustreError: 159457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306647 with bad export cookie 11547332499348916022 [ 4581.869472] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4581.876564] LustreError: 159457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4582.368567] LustreError: 163207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4582.369168] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.401521] Lustre: Skipped 1 previous similar message [ 4582.415382] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4586.400377] LustreError: 159472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.400396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4591.587263] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4591.587263] LustreError: 160628:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4596.192475] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4596.281924] Lustre: server umount lustre-MDT0001 complete [ 4604.667923] Lustre: server umount lustre-OST0000 complete [ 4610.575514] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 4616.529070] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 4636.381418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4636.518898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4636.534848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4636.590131] Lustre: lustre-MDT0000: new disk, initializing [ 4636.645782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4636.656869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4638.887723] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4645.436411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4645.513558] Lustre: 166081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4645.550587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4645.552704] Lustre: Skipped 1 previous similar message [ 4645.616175] Lustre: lustre-MDT0001: new disk, initializing [ 4645.620353] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4645.625290] Lustre: Skipped 1 previous similar message [ 4645.686406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4645.710665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4645.726419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4648.504746] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4655.463975] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4659.119209] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4663.139693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4663.297871] Lustre: lustre-OST0000: new disk, initializing [ 4663.300560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4663.303810] Lustre: 168050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4663.346748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4666.417523] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4668.945233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4668.950570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4669.017089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4672.517180] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4675.393894] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4675.567370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4677.956538] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4678.072280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4684.258556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4684.259078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.265706] LustreError: Skipped 1 previous similar message [ 4684.278111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4684.280927] Lustre: Skipped 1 previous similar message [ 4685.429500] Lustre: server umount lustre-OST0000 complete [ 4689.379312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.387212] Lustre: Skipped 2 previous similar messages [ 4702.177833] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4702.313618] Lustre: server umount lustre-MDT0000 complete [ 4704.738199] LustreError: 166087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4704.750971] LustreError: 166087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4704.808135] LustreError: 168049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306769 with bad export cookie 11547332499349074852 [ 4704.818749] LustreError: 168049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4704.830166] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4705.034899] Lustre: server umount lustre-MDT0001 complete [ 4710.823081] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:39:34 (1789306774) [ 4720.931357] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 4727.472418] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 4752.458325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4752.611285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4752.632701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4752.678344] Lustre: lustre-MDT0000: new disk, initializing [ 4752.734553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.751091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4755.221338] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4762.174476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4762.244434] Lustre: 172800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4762.422887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4765.068502] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4771.022322] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4774.522281] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4778.773045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4778.926523] Lustre: 174778:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4779.420568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4779.463231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4782.406700] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4788.500136] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4794.858843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4794.859238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4794.867241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4794.868634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4794.868639] Lustre: Skipped 6 previous similar messages [ 4794.885972] LustreError: Skipped 1 previous similar message [ 4798.378701] Lustre: server umount lustre-OST0000 complete [ 4803.555673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4806.452895] Lustre: server umount lustre-MDT0000 complete [ 4808.149356] LustreError: 172794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306873 with bad export cookie 11547332499349075734 [ 4808.149958] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4808.156418] LustreError: 172794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4808.314593] Lustre: server umount lustre-MDT0001 complete [ 4812.249559] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:41:16 (1789306876) [ 4819.049404] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 4823.761123] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 4842.450087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4842.600816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4842.620182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4842.623771] Lustre: Skipped 3 previous similar messages [ 4842.664939] Lustre: lustre-MDT0000: new disk, initializing [ 4842.667093] Lustre: Skipped 2 previous similar messages [ 4842.723954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.728579] Lustre: Skipped 2 previous similar messages [ 4842.740140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4842.743655] Lustre: Skipped 2 previous similar messages [ 4844.786431] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4850.505188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4850.570728] Lustre: 179092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4850.708421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4852.555076] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4857.441641] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4860.084285] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4863.783843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4863.940384] Lustre: 181066:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4866.798313] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4871.928731] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4873.792664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4883.940332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4883.940595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4883.948411] Lustre: Skipped 2 previous similar messages [ 4883.953917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4883.958532] Lustre: Skipped 5 previous similar messages [ 4886.234324] Lustre: server umount lustre-OST0000 complete [ 4902.368622] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4902.451878] Lustre: server umount lustre-MDT0000 complete [ 4904.242220] LustreError: 179083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306969 with bad export cookie 11547332499349076819 [ 4904.246223] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4904.247828] LustreError: 179083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4904.429095] Lustre: server umount lustre-MDT0001 complete [ 4909.870537] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 4911.350476] Key type lgssc unregistered [ 4911.521358] LNet: 182926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4911.524190] LNetError: 182926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4911.538347] LNet: Removed LNI 192.168.202.111@tcp [ 4912.000756] Key type .llcrypt unregistered [ 4912.002353] Key type ._llcrypt unregistered [ 4925.088029] Key type ._llcrypt registered [ 4925.089422] Key type .llcrypt registered [ 4925.133414] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 4925.640262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4925.659511] alg: No test for adler32 (adler32-zlib) [ 4926.584413] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 4926.726040] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 4928.336188] Key type lgssc registered [ 4928.984864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4933.180078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4933.193526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4934.389207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4936.500371] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4941.079853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4941.270968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4943.363559] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4947.872385] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4950.903959] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4954.835066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4955.082571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4958.023343] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4959.211397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4962.077884] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4979.687153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4979.697701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4979.714558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4981.730061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4981.733244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4981.737480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4985.553451] Lustre: server umount lustre-OST0000 complete [ 4991.973574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4991.974373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4991.978608] Lustre: Skipped 1 previous similar message [ 4991.986499] Lustre: Skipped 2 previous similar messages [ 4993.578750] Lustre: server umount lustre-MDT0000 complete [ 4995.469666] LustreError: 184296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307060 with bad export cookie 18102688074670306570 [ 4995.471511] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4995.475118] LustreError: 184296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4995.639366] Lustre: server umount lustre-MDT0001 complete [ 5001.082475] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 5002.658924] Key type lgssc unregistered [ 5002.861769] LNet: 187864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5002.865985] LNetError: 187864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5002.873842] LNet: Removed LNI 192.168.202.111@tcp [ 5003.327491] Key type .llcrypt unregistered [ 5003.329080] Key type ._llcrypt unregistered [ 5014.963872] Key type ._llcrypt registered [ 5014.965348] Key type .llcrypt registered [ 5015.007279] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 5015.352888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5015.377946] alg: No test for adler32 (adler32-zlib) [ 5016.261866] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 5016.374887] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 5017.968219] Key type lgssc registered [ 5018.435375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5021.719306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5021.729295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5022.917606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5024.610556] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5028.586670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5028.749542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5030.420225] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5033.857466] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5037.011680] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5040.618885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5040.810611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5044.488638] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5047.983611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5048.801752] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5058.530396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5058.531432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5058.539539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5058.541337] LustreError: Skipped 1 previous similar message [ 5058.544893] Lustre: Skipped 1 previous similar message [ 5059.558648] Lustre: server umount lustre-OST0000 complete [ 5063.061734] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff968384f6d800.ost_server_uuid 50 [ 5079.774762] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5090.274163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5090.279583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5090.284493] Lustre: Skipped 1 previous similar message [ 5090.287094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5093.345724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5093.347934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.354877] Lustre: server umount lustre-MDT0000 complete [ 5098.176047] LustreError: 189236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307163 with bad export cookie 6662367650799844330 [ 5098.182324] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5098.182410] LustreError: 189236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5098.335804] Lustre: server umount lustre-MDT0001 complete [ 5102.418071] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:46:06 (1789307166) [ 5105.565088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5105.802319] LustreError: 193165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5105.808774] LustreError: 193165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5105.842620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5107.808980] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5111.265734] LustreError: 193166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5111.754181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5111.920237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5113.591605] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5116.840723] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5119.163485] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5122.040117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5122.186214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5124.768546] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5128.322622] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5130.730906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5131.595757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5131.752765] Lustre: lustre-OST0001: new disk, initializing [ 5131.757196] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5131.763187] Lustre: 196006:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5134.673561] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5136.883788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5136.886959] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5136.907918] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5139.217510] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5140.961961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5140.964942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5140.970429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5140.972368] Lustre: Skipped 1 previous similar message [ 5145.069065] Lustre: lustre-OST0000: Not available for connect from 192.168.202.11@tcp (stopping) [ 5145.076000] Lustre: Skipped 2 previous similar messages [ 5146.950124] Lustre: server umount lustre-OST0000 complete [ 5147.110786] LustreError: 195902:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5147.117979] LustreError: 195902:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5149.847476] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5149.956920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5150.165942] LustreError: 194984:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5152.027059] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5152.122132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5155.292677] LustreError: 194987:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5155.299402] LustreError: 194987:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5167.588511] LustreError: 194984:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5167.588963] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5167.589037] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5167.589042] Lustre: Skipped 1 previous similar message [ 5167.590664] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5167.595820] LustreError: 194984:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 5167.601366] LustreError: Skipped 1 previous similar message [ 5168.938850] Lustre: server umount lustre-OST0001 complete [ 5176.758755] Lustre: server umount lustre-MDT0000 complete [ 5178.618414] LustreError: 193146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307243 with bad export cookie 6662367650799845387 [ 5178.619084] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5178.626445] LustreError: 193146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5178.783867] Lustre: server umount lustre-MDT0001 complete [ 5190.738529] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:47:34 (1789307254) [ 5194.122506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5194.279498] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5194.300980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5194.396573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5194.403074] Lustre: Skipped 1 previous similar message [ 5196.331643] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5200.713331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.761616] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5200.777530] Lustre: 199418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5200.816433] LustreError: 199429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5200.823226] LustreError: 199429:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5202.682162] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5205.968156] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5208.542740] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5211.557190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5211.656506] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5213.703449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5214.634729] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5217.989087] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5221.278437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5221.318051] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5224.580254] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5227.580756] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5228.114850] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5232.614087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5232.624232] Lustre: Skipped 3 previous similar messages [ 5232.630786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5232.634159] Lustre: Skipped 4 previous similar messages [ 5235.471239] Lustre: server umount lustre-OST0000 complete [ 5237.732808] LustreError: 201266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5237.738297] LustreError: 201266:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5247.971870] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5247.975408] LustreError: Skipped 1 previous similar message [ 5252.715204] Lustre: server umount lustre-OST0001 complete [ 5257.184892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5257.187464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5257.191353] Lustre: Skipped 3 previous similar messages [ 5260.358271] Lustre: server umount lustre-MDT0000 complete [ 5262.156484] LustreError: 200163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307327 with bad export cookie 6662367650799846787 [ 5262.159598] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5262.161552] LustreError: 200163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5262.325967] Lustre: server umount lustre-MDT0001 complete [ 5272.856455] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:48:56 (1789307336) [ 5280.155983] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 5285.163604] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 5301.705472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5301.839600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5301.855644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5301.893720] Lustre: lustre-MDT0000: new disk, initializing [ 5301.933442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5301.937709] Lustre: Skipped 3 previous similar messages [ 5301.947054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5303.616046] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5309.062683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5309.112611] Lustre: 207491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5309.146139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5309.148504] Lustre: Skipped 1 previous similar message [ 5309.187698] Lustre: lustre-MDT0001: new disk, initializing [ 5309.247659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5309.251715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5311.043840] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5316.106333] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5319.009933] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5322.150180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5322.288845] Lustre: lustre-OST0000: new disk, initializing [ 5322.290974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5322.294710] Lustre: 209457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5324.994167] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5328.388587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5328.393228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5328.429531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5329.441469] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5331.416070] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5331.521133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5333.197048] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5333.288649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5338.594782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5338.596671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5338.599425] Lustre: Skipped 1 previous similar message [ 5338.599959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5338.602870] LustreError: Skipped 1 previous similar message [ 5338.609078] Lustre: Skipped 7 previous similar messages [ 5340.312744] Lustre: server umount lustre-OST0000 complete [ 5348.070198] Lustre: server umount lustre-MDT0000 complete [ 5348.832922] LustreError: 207503:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5348.841367] LustreError: 207503:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 5349.774100] LustreError: 207483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307414 with bad export cookie 6662367650799848572 [ 5349.778109] LustreError: 207483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5349.779054] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5349.907774] Lustre: server umount lustre-MDT0001 complete [ 5353.404140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5355.252241] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5358.777215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5360.448692] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5363.383816] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5366.161884] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5369.246651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5369.448815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5369.452554] Lustre: Skipped 4 previous similar messages [ 5372.186388] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5375.794545] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5377.693616] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5378.825062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5380.590616] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5385.786184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 5 sec [ 5389.280931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5392.621598] Lustre: server umount lustre-OST0000 complete [ 5395.047348] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5395.141500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5396.792686] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5420.768108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5423.105184] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5425.887948] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5426.517095] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5430.562950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5430.565411] Lustre: Skipped 5 previous similar messages [ 5433.658986] Lustre: server umount lustre-OST0000 complete [ 5435.172381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5435.180699] Lustre: Skipped 6 previous similar messages [ 5443.313735] Lustre: server umount lustre-MDT0000 complete [ 5444.782304] LustreError: 211652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307509 with bad export cookie 6662367650799849440 [ 5444.785109] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5444.785955] LustreError: 211652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5444.915393] Lustre: server umount lustre-MDT0001 complete [ 5448.682376] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:51:52 (1789307512) [ 5451.580657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5453.592960] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5456.961383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5458.526712] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5460.986223] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5462.894705] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5465.225895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5467.563718] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5470.272392] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5471.772300] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5476.944285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 5478.292668] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5478.360531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5480.555913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5480.626024] Lustre: lustre-OST0001: new disk, initializing [ 5480.627849] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5480.630718] Lustre: 221086:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5482.613783] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5486.361873] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5487.705770] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5490.690672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5490.694842] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5490.732216] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5490.852906] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5492.252776] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5492.325331] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5495.780766] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5495.783380] LustreError: Skipped 2 previous similar messages [ 5499.073737] Lustre: server umount lustre-OST0001 complete [ 5500.900582] LustreError: 219503:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5500.904288] LustreError: 219503:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 5500.929214] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5500.992112] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5502.212987] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5502.273746] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5526.014245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5526.088452] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5526.091357] Lustre: Skipped 5 previous similar messages [ 5528.016092] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5530.429626] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5530.943155] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5531.116495] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5531.120702] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5531.120903] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5531.125774] Lustre: Skipped 1 previous similar message [ 5537.723040] Lustre: server umount lustre-OST0001 complete [ 5561.828091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5561.829836] Lustre: Skipped 14 previous similar messages [ 5564.384088] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5564.428464] Lustre: server umount lustre-OST0000 complete [ 5566.945750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5566.951141] Lustre: Skipped 6 previous similar messages [ 5571.930317] Lustre: server umount lustre-MDT0000 complete [ 5573.114354] LustreError: 217712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307638 with bad export cookie 6662367650799850371 [ 5573.119340] LustreError: 217712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5573.120219] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5579.318607] Lustre: server umount lustre-MDT0001 complete [ 5587.485969] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:54:11 (1789307651) [ 5589.783481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5589.879343] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5589.890738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5591.342866] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5594.268973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5594.319031] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5594.329127] Lustre: 226340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5595.849644] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5598.460134] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5600.316103] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5602.621989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5602.686623] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5604.716184] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5607.324404] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5616.074272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5616.102097] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5618.016687] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5620.391602] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5621.914490] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5623.221564] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5623.282753] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5624.586064] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5624.659572] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5626.420950] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff96839833b800.ost_server_uuid 50 [ 5626.962115] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff96839833b800.ost_server_uuid in FULL state after 0 sec [ 5627.308935] Lustre: Permanently deactivating lustre-OST0001 [ 5627.314236] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5628.027870] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5628.122059] Lustre: Permanently reactivating lustre-OST0001 [ 5635.265690] Lustre: server umount lustre-OST0001 complete [ 5636.576997] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5636.580102] LustreError: Skipped 3 previous similar messages [ 5642.640534] Lustre: server umount lustre-OST0000 complete [ 5653.984153] LustreError: 230913:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 5654.012934] Lustre: server umount lustre-MDT0000 complete [ 5655.321886] LustreError: 227083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307720 with bad export cookie 6662367650799851652 [ 5655.324492] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5655.326973] LustreError: 227083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5655.435725] Lustre: server umount lustre-MDT0001 complete [ 5663.053440] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:55:27 (1789307727) [ 5665.632966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.729559] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5665.739551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5665.743090] Lustre: Skipped 5 previous similar messages [ 5667.161193] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5669.985868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5670.021419] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5670.031509] Lustre: 233034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5671.542731] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5674.045625] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5675.904696] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5678.262193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5678.339648] Lustre: Permanently deactivating lustre-OST0000 [ 5678.345059] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5680.113684] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5680.116665] Lustre: Skipped 1 previous similar message [ 5680.118034] LustreError: 235201:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 5680.122098] LustreError: 235201:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 5680.298976] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5682.886834] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5685.298038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5685.506248] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5687.461461] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5690.217566] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5690.901887] Lustre: Permanently reactivating lustre-OST0000 [ 5693.444774] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5693.447704] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.451815] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5693.466676] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5693.474692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5709.385370] Lustre: server umount lustre-OST0001 complete [ 5725.152118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5725.202839] Lustre: server umount lustre-OST0000 complete [ 5732.761792] Lustre: server umount lustre-MDT0000 complete [ 5733.974044] LustreError: 233025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307799 with bad export cookie 6662367650799853388 [ 5733.974723] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5733.978195] LustreError: 233025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5734.100271] Lustre: server umount lustre-MDT0001 complete [ 5737.608329] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 5738.612558] Key type lgssc unregistered [ 5738.741323] LNet: 237906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5738.743697] LNetError: 237906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5738.751304] LNet: Removed LNI 192.168.202.111@tcp [ 5739.060673] Key type .llcrypt unregistered [ 5739.062233] Key type ._llcrypt unregistered [ 5747.563919] Key type ._llcrypt registered [ 5747.565944] Key type .llcrypt registered [ 5747.602245] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 5747.920485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5747.927401] alg: No test for adler32 (adler32-zlib) [ 5748.776370] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 5748.862951] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 5750.448194] Key type lgssc registered [ 5750.828114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5754.123475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5754.132660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5755.295074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5756.622694] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5759.607360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5759.661377] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5759.673967] Lustre: 239335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5759.678371] Lustre: Permanently deactivating lustre-MDT0001 [ 5759.685210] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5759.802405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5761.205536] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5761.519675] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5761.522740] LustreError: 239376:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5761.526921] LustreError: 239376:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5761.530320] LustreError: 239376:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 5763.648568] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5765.477120] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5767.732330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5767.840739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5768.868179] LustreError: 241185:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.870855] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5768.874336] LustreError: 241185:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5769.831470] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5772.400592] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5774.308917] LustreError: 241185:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.895485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5774.976086] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5777.069857] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5779.810571] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5780.144646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5780.144791] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5780.146541] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5785.508981] Lustre: Permanently reactivating lustre-MDT0001 [ 5785.517292] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5785.520613] Lustre: Skipped 2 previous similar messages [ 5785.971563] Lustre: lustre-MDT0001: Client 0af8aa17-ec09-49cb-87a2-0c3fb9e77bfb (at 192.168.202.11@tcp) reconnecting [ 5788.713959] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5789.346451] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5791.400067] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5791.485165] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5792.965224] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5793.033647] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5793.942638] Lustre: Permanently deactivating lustre-MDT0001 [ 5793.949729] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5793.952710] Lustre: Skipped 2 previous similar messages [ 5800.970976] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5811.172359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5811.172836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5811.176785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5811.179462] Lustre: Skipped 2 previous similar messages [ 5811.181362] Lustre: Skipped 1 previous similar message [ 5813.606934] Lustre: server umount lustre-MDT0000 complete [ 5814.912391] LustreError: 242891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307880 with bad export cookie 11911634959825897273 [ 5814.916787] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5814.919609] LustreError: 242891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5815.032084] Lustre: server umount lustre-MDT0001 complete [ 5822.465449] Lustre: server umount lustre-OST0000 complete [ 5826.589441] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:58:10 (1789307890) [ 5830.494220] Lustre: server umount lustre-OST0001 complete [ 5832.809484] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 5836.118598] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 5848.151952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5848.238866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5848.243024] Lustre: Skipped 2 previous similar messages [ 5848.251392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5848.288461] Lustre: lustre-MDT0000: new disk, initializing [ 5848.311678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5848.318686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5849.783776] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5854.191594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5854.235981] Lustre: 247688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5854.260705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5854.263505] Lustre: Skipped 1 previous similar message [ 5854.300190] Lustre: lustre-MDT0001: new disk, initializing [ 5854.342462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5854.346794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5855.757792] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5859.600719] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5861.576197] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5863.897075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5863.997995] Lustre: lustre-OST0000: new disk, initializing [ 5864.000488] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5864.003435] Lustre: 249657:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5864.034327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5864.036578] Lustre: Skipped 1 previous similar message [ 5866.165031] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5870.259986] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5872.632125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5872.635344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5872.670320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5876.224610] LustreError: 247695:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5880.498673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5880.562208] LustreError: 250375:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5880.573951] Lustre: lustre-OST0001: new disk, initializing [ 5880.576386] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5880.579874] Lustre: 250773:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5880.606823] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5882.593667] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5885.940139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5885.943211] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5885.955363] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400 [ 5886.320127] LustreError: 247695:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5886.520927] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5888.047285] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5888.405108] LustreError: 247695:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5898.504130] LustreError: 247695:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5900.261689] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5900.268918] Lustre: Skipped 1 previous similar message [ 5900.273119] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5900.273854] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5900.278702] Lustre: Skipped 3 previous similar messages [ 5902.813442] Lustre: lustre-OST0001: Not available for connect from 192.168.202.11@tcp (stopping) [ 5905.379377] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5905.381942] Lustre: Skipped 1 previous similar message [ 5905.468344] Lustre: server umount lustre-OST0001 complete [ 5910.500930] LustreError: 249643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5910.505215] LustreError: 249643:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5915.620497] LustreError: 249643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5915.628444] LustreError: 249643:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5918.688923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5918.691288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5918.694981] Lustre: Skipped 1 previous similar message [ 5918.696928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5918.698869] Lustre: Skipped 1 previous similar message [ 5923.538491] Lustre: server umount lustre-OST0000 complete [ 5925.857584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5925.861386] Lustre: Skipped 2 previous similar messages [ 5930.977208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5930.980137] Lustre: Skipped 5 previous similar messages [ 5939.168113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5939.231800] Lustre: server umount lustre-MDT0000 complete [ 5940.374145] LustreError: 247683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308005 with bad export cookie 11911634959825900087 [ 5940.377644] LustreError: 247683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5940.378173] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5946.549786] Lustre: server umount lustre-MDT0001 complete [ 5950.447668] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 5951.512834] Key type lgssc unregistered [ 5951.660298] LNet: 252821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5951.663186] LNetError: 252821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5951.673322] LNet: Removed LNI 192.168.202.111@tcp [ 5951.987112] Key type .llcrypt unregistered [ 5951.988783] Key type ._llcrypt unregistered [ 5966.645374] Key type ._llcrypt registered [ 5966.646982] Key type .llcrypt registered [ 5966.687425] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 5967.039684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5967.070033] alg: No test for adler32 (adler32-zlib) [ 5967.928844] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 5968.024851] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 5969.608192] Key type lgssc registered [ 5969.991051] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5972.450121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5972.455956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5973.535578] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5973.541849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5973.591519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5974.852784] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5977.473911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.503518] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5977.513949] Lustre: 255002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5977.603832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5978.815399] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5981.293466] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5983.208479] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5985.644960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5985.712155] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5985.769619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5987.405281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5987.516717] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5989.799857] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6033.378525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6033.380404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6033.381215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6033.381449] LustreError: Skipped 1 previous similar message [ 6033.387044] Lustre: Skipped 1 previous similar message [ 6034.521458] Lustre: server umount lustre-OST0000 complete [ 6035.937665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6039.790526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6039.892745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.627977] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6043.848768] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6053.156854] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.161149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.161166] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6053.165738] Lustre: Skipped 1 previous similar message [ 6078.945474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6078.947347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.948283] LustreError: Skipped 1 previous similar message [ 6078.952250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6078.954593] Lustre: Skipped 2 previous similar messages [ 6083.166888] Lustre: server umount lustre-OST0000 complete [ 6085.088981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6085.092545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6085.096376] Lustre: Skipped 1 previous similar message [ 6085.098106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6089.186138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6089.186389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.594233] Lustre: server umount lustre-MDT0000 complete [ 6091.949466] LustreError: 256858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308157 with bad export cookie 2678445715056900969 [ 6091.952972] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6091.954264] LustreError: 256858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6092.112510] Lustre: server umount lustre-MDT0001 complete [ 6095.895625] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 6096.952068] Key type lgssc unregistered [ 6097.081389] LNet: 260422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6097.084501] LNetError: 260422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6097.095376] LNet: Removed LNI 192.168.202.111@tcp [ 6097.414907] Key type .llcrypt unregistered [ 6097.416777] Key type ._llcrypt unregistered [ 6106.375219] Key type ._llcrypt registered [ 6106.376204] Key type .llcrypt registered [ 6106.417881] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6106.927965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6106.938972] alg: No test for adler32 (adler32-zlib) [ 6107.797777] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 6107.885790] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 6109.464178] Key type lgssc registered [ 6109.879491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6112.540304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6112.547194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6113.704889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6115.107119] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6117.946540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6118.076385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6119.478403] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6121.876382] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6123.697876] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6126.592647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6126.719919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6128.821139] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6134.903179] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6138.024954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6148.579124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.580138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6148.582717] Lustre: Skipped 1 previous similar message [ 6148.586223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6153.697814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6153.700013] Lustre: Skipped 1 previous similar message [ 6154.265939] Lustre: server umount lustre-OST0000 complete [ 6158.817560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6158.818682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6158.822550] Lustre: Skipped 1 previous similar message [ 6158.825547] Lustre: Skipped 1 previous similar message [ 6161.695241] Lustre: server umount lustre-MDT0000 complete [ 6162.983528] LustreError: 263651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308228 with bad export cookie 10670827319048912249 [ 6162.986620] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6162.988690] LustreError: 263651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6163.106477] Lustre: server umount lustre-MDT0001 complete [ 6166.975382] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 6168.017821] Key type lgssc unregistered [ 6168.150378] LNet: 266110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6168.153379] LNetError: 266110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6168.166308] LNet: Removed LNI 192.168.202.111@tcp [ 6168.493956] Key type .llcrypt unregistered [ 6168.494946] Key type ._llcrypt unregistered [ 6175.364541] Key type ._llcrypt registered [ 6175.366571] Key type .llcrypt registered [ 6175.405182] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6175.729826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6175.757136] alg: No test for adler32 (adler32-zlib) [ 6176.605133] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 6176.691946] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 6178.264192] Key type lgssc registered [ 6178.614989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6181.096215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6181.101929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6182.236455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6183.472663] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6186.228258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6186.344426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6187.613921] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6189.857858] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6191.559685] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6193.690849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6193.807327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6195.573774] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6197.843672] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6207.143686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6212.578114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6212.578370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6212.578899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6212.582571] Lustre: Skipped 1 previous similar message [ 6214.105176] Lustre: server umount lustre-OST0000 complete [ 6217.185098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6217.187834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6217.192186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6217.194052] Lustre: Skipped 1 previous similar message [ 6217.497410] Lustre: server umount lustre-MDT0000 complete [ 6217.697518] LustreError: 268037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6217.702544] LustreError: 268037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6218.691073] LustreError: 267285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308283 with bad export cookie 11031384249972152179 [ 6218.693792] LustreError: 267285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6218.694805] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6218.808325] Lustre: server umount lustre-MDT0001 complete [ 6222.201337] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 6223.133893] Key type lgssc unregistered [ 6223.250535] LNet: 270997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6223.253810] LNetError: 270997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6223.262278] LNet: Removed LNI 192.168.202.111@tcp [ 6223.578317] Key type .llcrypt unregistered [ 6223.579814] Key type ._llcrypt unregistered [ 6232.223420] Key type ._llcrypt registered [ 6232.224943] Key type .llcrypt registered [ 6232.260032] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6232.610859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6232.647842] alg: No test for adler32 (adler32-zlib) [ 6233.498991] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 6233.581705] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 6235.160161] Key type lgssc registered [ 6235.513035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6237.992837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6237.999885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6239.128795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.309261] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6243.035593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6243.151240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6244.420346] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6246.797214] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6248.687829] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6250.962863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6251.078768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6252.950735] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6255.384652] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6264.423042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6274.017465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6274.019490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6274.023046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6275.041900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6275.041915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6275.042661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6279.185430] Lustre: server umount lustre-OST0000 complete [ 6284.256701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6284.258751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6284.262068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6284.263493] Lustre: Skipped 1 previous similar message [ 6286.549679] Lustre: server umount lustre-MDT0000 complete [ 6287.770454] LustreError: 274227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308352 with bad export cookie 10681762684471511690 [ 6287.774416] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6287.774706] LustreError: 274227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6287.891737] Lustre: server umount lustre-MDT0001 complete [ 6291.477345] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 6292.485830] Key type lgssc unregistered [ 6292.595543] LNet: 276711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6292.598292] LNetError: 276711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6292.606407] LNet: Removed LNI 192.168.202.111@tcp [ 6292.907901] Key type .llcrypt unregistered [ 6292.909239] Key type ._llcrypt unregistered [ 6299.941528] Key type ._llcrypt registered [ 6299.942458] Key type .llcrypt registered [ 6299.980765] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6300.299203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6300.348832] alg: No test for adler32 (adler32-zlib) [ 6301.194383] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 6301.285683] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 6302.856202] Key type lgssc registered [ 6303.169634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6305.439947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6305.445554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6306.583303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6307.862304] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6310.694077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6310.817060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6312.194187] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6314.494687] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6316.215435] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6318.367664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6318.467171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6320.197708] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6322.391528] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6331.814795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6336.994554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6336.995157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6336.997946] Lustre: Skipped 1 previous similar message [ 6337.001398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6338.451144] Lustre: server umount lustre-OST0000 complete [ 6341.600844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6341.603484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6341.607954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6341.610294] Lustre: Skipped 1 previous similar message [ 6345.889329] Lustre: server umount lustre-MDT0000 complete [ 6347.171472] LustreError: 279784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308412 with bad export cookie 12379819963652819115 [ 6347.174738] LustreError: 279784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6347.175284] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6347.314480] Lustre: server umount lustre-MDT0001 complete [ 6350.954526] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 6351.995789] Key type lgssc unregistered [ 6352.133391] LNet: 281636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6352.135825] LNetError: 281636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6352.143320] LNet: Removed LNI 192.168.202.111@tcp [ 6352.459822] Key type .llcrypt unregistered [ 6352.461210] Key type ._llcrypt unregistered [ 6356.332513] Key type ._llcrypt registered [ 6356.333610] Key type .llcrypt registered [ 6356.375895] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6362.010414] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 6368.032591] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6368.369537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6368.405034] alg: No test for adler32 (adler32-zlib) [ 6369.258472] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 6369.351409] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 6370.928207] Key type lgssc registered [ 6371.313783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6381.878836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6381.883699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6382.956819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6382.966753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6382.998986] Lustre: lustre-MDT0000: new disk, initializing [ 6383.017450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6383.023862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6384.243184] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6388.085168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6388.124423] Lustre: 285155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6388.144308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6388.145857] Lustre: Skipped 1 previous similar message [ 6388.177863] Lustre: lustre-MDT0001: new disk, initializing [ 6388.205921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6388.216294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6388.219330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6389.453228] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6393.050146] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6394.839206] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6397.060989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6397.156253] Lustre: lustre-OST0000: new disk, initializing [ 6397.158914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6397.161494] Lustre: 287122:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6397.187850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6398.840421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6398.844769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6398.871332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6399.075174] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6402.650513] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6403.956187] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6404.038985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6405.301882] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6405.365330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6409.185917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6409.187557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6409.188328] LustreError: Skipped 1 previous similar message [ 6409.188666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6409.191870] Lustre: Skipped 1 previous similar message [ 6412.048227] Lustre: server umount lustre-OST0000 complete [ 6413.792781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6413.794908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6413.798538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6413.800298] Lustre: Skipped 1 previous similar message [ 6419.416946] Lustre: server umount lustre-MDT0000 complete [ 6420.609278] LustreError: 285950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308485 with bad export cookie 14141434157843344400 [ 6420.612543] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6420.613605] LustreError: 285950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6420.715605] Lustre: server umount lustre-MDT0001 complete [ 6423.378906] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:08:07 (1789308487) [ 6425.518644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.665630] LustreError: 289532:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6425.671381] LustreError: 289532:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6425.689644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6426.959676] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6429.635504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6429.746093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6431.068317] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6433.335960] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6435.154050] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6437.363666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6439.285901] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6441.495519] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6455.265323] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6455.266961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6455.269806] Lustre: Skipped 1 previous similar message [ 6455.271375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6455.272674] Lustre: Skipped 2 previous similar messages [ 6461.004609] Lustre: server umount lustre-OST0000 complete [ 6464.994068] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6464.997516] LustreError: Skipped 1 previous similar message [ 6464.999780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6465.006294] Lustre: Skipped 1 previous similar message [ 6465.008734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6465.010913] Lustre: Skipped 2 previous similar messages [ 6468.436324] Lustre: server umount lustre-MDT0000 complete [ 6469.686204] LustreError: 289513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308534 with bad export cookie 14141434157843345268 [ 6469.688552] LustreError: 290271:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 6469.689603] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6469.693222] LustreError: 289513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6469.817525] Lustre: server umount lustre-MDT0001 complete [ 6473.504525] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 6474.545310] Key type lgssc unregistered [ 6474.676414] LNet: 293084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6474.679342] LNetError: 293084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6474.689303] LNet: Removed LNI 192.168.202.111@tcp [ 6475.013554] Key type .llcrypt unregistered [ 6475.015149] Key type ._llcrypt unregistered [ 6483.656623] Key type ._llcrypt registered [ 6483.657598] Key type .llcrypt registered [ 6483.695451] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 6489.923029] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6490.349968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6490.390653] alg: No test for adler32 (adler32-zlib) [ 6491.246073] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 6491.333044] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 6492.912168] Key type lgssc registered [ 6493.273587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6508.570073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6508.578640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6509.662053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6509.674038] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6509.711635] Lustre: lustre-MDT0000: new disk, initializing [ 6509.732485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6509.737991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6511.146600] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6515.423950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6515.466515] Lustre: 296876:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6515.492273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6515.494403] Lustre: Skipped 1 previous similar message [ 6515.532879] Lustre: lustre-MDT0001: new disk, initializing [ 6515.569610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6515.585686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6515.591251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6517.046466] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6520.673623] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6522.434474] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6524.543176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6524.638583] Lustre: lustre-OST2710: new disk, initializing [ 6524.640247] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6524.642601] Lustre: 298843:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6524.668793] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6526.396179] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6529.892631] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6532.014345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6532.073166] Lustre: lustre-OST03e8: new disk, initializing [ 6532.074661] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6532.076932] Lustre: 299896:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6532.108143] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6533.958194] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6534.082556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6534.086704] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6534.089561] Lustre: Skipped 1 previous similar message [ 6534.131980] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6534.136133] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6536.192687] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6538.636435] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6538.695097] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6539.890190] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6539.950235] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6541.273098] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6541.354099] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6542.704048] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6542.774166] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6544.356246] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6544.357571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6544.360230] Lustre: Skipped 3 previous similar messages [ 6544.363947] Lustre: Skipped 3 previous similar messages [ 6549.473268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6549.921800] Lustre: server umount lustre-MDT0000 complete [ 6551.129204] LustreError: 301642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308616 with bad export cookie 14841602702221732377 [ 6551.132796] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6551.134371] LustreError: 301642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6551.253477] Lustre: server umount lustre-MDT0001 complete [ 6562.825453] Lustre: server umount lustre-OST2710 complete [ 6574.373184] Lustre: server umount lustre-OST03e8 complete [ 6580.286794] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 6583.210623] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6594.741710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.855831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6594.867799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6594.910157] Lustre: lustre-MDT0000: new disk, initializing [ 6594.938618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6594.948706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6594.952816] Lustre: Skipped 1 previous similar message [ 6596.413020] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6600.702665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6600.743314] Lustre: 305145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6600.845127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6600.848241] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6602.160631] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6605.898693] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6607.711344] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6609.948575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6610.054603] Lustre: lustre-OST0000: new disk, initializing [ 6610.056164] Lustre: Skipped 1 previous similar message [ 6610.057557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6610.059460] Lustre: Skipped 2 previous similar messages [ 6610.061325] Lustre: 307112:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6610.090372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6610.093933] Lustre: Skipped 1 previous similar message [ 6612.019131] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6612.027304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6612.030833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6612.070811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6615.775691] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6617.206668] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6617.275643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6618.615819] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6618.685755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6622.179570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6622.180629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6622.184130] Lustre: Skipped 1 previous similar message [ 6622.185995] Lustre: Skipped 3 previous similar messages [ 6622.187479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6625.422944] Lustre: server umount lustre-OST0000 complete [ 6626.785083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6626.787383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6626.791811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6632.417675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6632.419604] Lustre: Skipped 3 previous similar messages [ 6632.858226] Lustre: server umount lustre-MDT0000 complete [ 6634.091328] LustreError: 305136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308699 with bad export cookie 14841602702221733959 [ 6634.094514] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6634.094917] LustreError: 305136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6634.224185] Lustre: server umount lustre-MDT0001 complete [ 6636.981455] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:11:41 (1789308701) [ 6637.533188] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6638.080427] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:11:42 (1789308702) [ 6639.185468] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6648.132168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6648.218445] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6648.225072] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6648.284960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6649.612350] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6652.447888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6652.483518] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6652.492792] Lustre: 310964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6653.991123] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6669.794992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6669.862350] Lustre: Denying initial registration attempt from nid 192.168.202.111@tcp, specified as failover [ 6669.864854] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6669.867852] LustreError: 312412:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 6669.870268] LustreError: 312412:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6669.872275] LustreError: 312412:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6669.895583] Lustre: server umount lustre-OST0000 complete [ 6669.898218] LustreError: 312412:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6673.377613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6673.378046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6673.384058] Lustre: Skipped 1 previous similar message [ 6673.386177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6673.387981] Lustre: Skipped 1 previous similar message [ 6678.235933] Lustre: server umount lustre-MDT0000 complete [ 6679.553247] LustreError: 311706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308744 with bad export cookie 14841602702221734827 [ 6679.556820] LustreError: 311706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6679.557513] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6679.665541] Lustre: server umount lustre-MDT0001 complete [ 6683.406644] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 6684.296706] Key type lgssc unregistered [ 6684.417395] LNet: 313536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6684.419831] LNetError: 313536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6684.428236] LNet: Removed LNI 192.168.202.111@tcp [ 6684.707321] Key type .llcrypt unregistered [ 6684.708196] Key type ._llcrypt unregistered [ 6687.902979] Key type ._llcrypt registered [ 6687.904284] Key type .llcrypt registered [ 6687.936086] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6688.235314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6688.250926] alg: No test for adler32 (adler32-zlib) [ 6689.094345] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 6689.179826] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 6690.760175] Key type lgssc registered [ 6691.062408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6714.747123] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6717.630122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6717.635672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6718.717208] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6718.725966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6718.780067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6720.162082] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6723.001060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6723.032483] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6723.042642] Lustre: 316271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6723.133918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6724.409864] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6726.758869] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6728.567380] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6743.843038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6743.915647] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6743.979208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6745.816988] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6748.162045] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6748.865783] Lustre: server umount lustre-OST0000 complete [ 6753.761198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6753.763741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6753.768795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6756.182612] Lustre: server umount lustre-MDT0000 complete [ 6757.438759] LustreError: 316262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308822 with bad export cookie 15404735846022703769 [ 6757.441954] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6757.443140] LustreError: 316262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6757.563827] Lustre: server umount lustre-MDT0001 complete [ 6761.341520] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 6762.384879] Key type lgssc unregistered [ 6762.507294] LNet: 319831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6762.509679] LNetError: 319831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6762.520299] LNet: Removed LNI 192.168.202.111@tcp [ 6762.820807] Key type .llcrypt unregistered [ 6762.821878] Key type ._llcrypt unregistered [ 6771.911837] Key type ._llcrypt registered [ 6771.912922] Key type .llcrypt registered [ 6771.949966] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6772.386873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6772.392932] alg: No test for adler32 (adler32-zlib) [ 6773.245879] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 6773.328841] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 6774.904135] Key type lgssc registered [ 6775.293211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6777.854302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6777.859712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6778.999431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6780.397107] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6783.349287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6783.472155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6784.851673] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6787.377047] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6789.210290] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6793.443512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6793.563747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6795.485665] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6797.895171] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6813.659412] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.11@tcp (stopping) [ 6814.177212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6814.179942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.184888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6815.202267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6815.204377] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6815.208382] Lustre: Skipped 1 previous similar message [ 6816.670379] Lustre: server umount lustre-MDT0000 complete [ 6817.954521] LustreError: 321203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308883 with bad export cookie 48047119564680521 [ 6817.959475] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6818.772693] LustreError: 323711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6818.780317] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.11@tcp (stopping) [ 6818.780641] LustreError: 323711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6818.782762] Lustre: Skipped 2 previous similar messages [ 6820.320889] LustreError: 321215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6820.321152] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6822.368803] LustreError: 323711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6823.893251] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.11@tcp (stopping) [ 6823.895593] Lustre: Skipped 2 previous similar messages [ 6824.126367] Lustre: server umount lustre-MDT0001 complete [ 6826.306198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6829.121316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6843.808589] LustreError: 320530:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009460c37f x1876226304140032/t0(0) o250->MGC192.168.202.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6843.917372] LustreError: 323052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6843.924718] LustreError: 323052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6843.942216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6843.958884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6845.241237] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6848.042540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.164407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6849.486439] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6851.871721] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6852.266223] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6854.405790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6856.158671] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6856.719586] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6858.721329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6858.724693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6858.729797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6864.015295] Lustre: server umount lustre-OST0000 complete [ 6867.939255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6867.943385] Lustre: Skipped 2 previous similar messages [ 6871.325898] Lustre: server umount lustre-MDT0000 complete [ 6872.531664] LustreError: 321203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308937 with bad export cookie 48047119564687675 [ 6872.535472] LustreError: 321203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6872.536513] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6872.671351] Lustre: server umount lustre-MDT0001 complete [ 6876.124409] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 6877.093149] Key type lgssc unregistered [ 6877.227358] LNet: 327499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6877.229470] LNetError: 327499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6877.242328] LNet: Removed LNI 192.168.202.111@tcp [ 6877.556118] Key type .llcrypt unregistered [ 6877.557677] Key type ._llcrypt unregistered [ 6886.394652] Key type ._llcrypt registered [ 6886.395564] Key type .llcrypt registered [ 6886.432070] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6886.761738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6886.783070] alg: No test for adler32 (adler32-zlib) [ 6887.631345] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 6887.716863] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 6889.296234] Key type lgssc registered [ 6889.639964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6892.145307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6892.151288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6893.298490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6894.697329] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6897.489479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6897.636171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6898.954822] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6901.286852] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6903.131469] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6907.169765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6907.173905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6907.176923] Lustre: Skipped 1 previous similar message [ 6910.232590] Lustre: server umount lustre-MDT0000 complete [ 6911.390329] LustreError: 328870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308976 with bad export cookie 9760888268673309685 [ 6911.393580] LustreError: 328870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6911.393611] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6911.508495] Lustre: server umount lustre-MDT0001 complete [ 6916.156730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6916.249119] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6916.260456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6916.330671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6917.632546] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6920.399921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6920.432460] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6920.441819] Lustre: 331474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6920.535355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6921.797508] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6924.165784] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6925.996956] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6930.787876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6930.862728] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6930.866327] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6930.871107] LustreError: 333359:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 6930.874378] LustreError: 333359:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6930.877435] LustreError: 333359:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6930.903338] Lustre: server umount lustre-OST0000 complete [ 6930.905566] LustreError: 333359:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6935.714988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6935.783441] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6935.855478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6937.678609] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6940.092568] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6942.728487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6942.752423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 6942.772653] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6942.809881] Lustre: lustre-OST0001: new disk, initializing [ 6942.811780] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6942.814886] Lustre: 334897:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6944.772142] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6947.143700] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6948.082598] Lustre: server umount lustre-OST0001 complete [ 6948.859614] LustreError: 333825:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6948.863747] LustreError: 333825:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6952.928963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6952.929220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6952.931108] LustreError: Skipped 1 previous similar message [ 6952.931591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6952.939370] Lustre: Skipped 2 previous similar messages [ 6953.957045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6953.958399] LustreError: 335898:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6953.959394] Lustre: Skipped 2 previous similar messages [ 6953.963343] LustreError: 335898:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6955.349843] Lustre: server umount lustre-OST0000 complete [ 6959.074944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.076513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6959.078687] Lustre: Skipped 1 previous similar message [ 6959.082935] Lustre: Skipped 1 previous similar message [ 6962.717911] Lustre: server umount lustre-MDT0000 complete [ 6963.987537] LustreError: 334896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789309029 with bad export cookie 9760888268673310224 [ 6963.991231] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6964.113557] Lustre: server umount lustre-MDT0001 complete [ 6967.900352] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 6969.003634] Key type lgssc unregistered [ 6969.152461] LNet: 336848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6969.154864] LNetError: 336848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6969.165467] LNet: Removed LNI 192.168.202.111@tcp [ 6969.520640] Key type .llcrypt unregistered [ 6969.522278] Key type ._llcrypt unregistered [ 6999.815311] Key type ._llcrypt registered [ 6999.816376] Key type .llcrypt registered [ 6999.853935] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 7005.965750] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 7006.431943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7006.461414] alg: No test for adler32 (adler32-zlib) [ 7007.333755] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 7007.428736] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 7009.016161] Key type lgssc registered [ 7009.392945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7020.861960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7020.867924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7021.946722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7021.957404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7021.992343] Lustre: lustre-MDT0000: new disk, initializing [ 7022.012590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7022.018354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7023.378321] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7027.517294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7027.557958] Lustre: 341745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7027.592212] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7027.594232] Lustre: Skipped 1 previous similar message [ 7027.628960] Lustre: lustre-MDT0001: new disk, initializing [ 7027.665925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7027.677791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7027.681183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7029.052037] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7032.784692] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7034.557573] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7036.810684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7036.938951] Lustre: lustre-OST0000: new disk, initializing [ 7036.940609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7036.943348] Lustre: 343712:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7036.979574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7039.017695] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7042.905799] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7043.580101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7043.583608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7043.627135] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7044.420908] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7044.506835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7046.006472] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7046.080089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7048.678533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7048.678633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7048.680027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7048.681880] Lustre: Skipped 1 previous similar message [ 7048.684905] LustreError: Skipped 1 previous similar message [ 7052.823479] Lustre: server umount lustre-OST0000 complete [ 7058.400914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7058.403380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7058.407919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7058.409525] Lustre: Skipped 1 previous similar message [ 7060.250923] Lustre: server umount lustre-MDT0000 complete [ 7061.465332] LustreError: 342541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789309126 with bad export cookie 5465153401618589311 [ 7061.468500] LustreError: 342541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7061.469853] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7061.589117] Lustre: server umount lustre-MDT0001 complete [ 7064.245320] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:18:48 (1789309128) [ 7066.696110] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:18:50 (1789309130) [ 7069.662572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.863934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7071.259776] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7074.083032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7074.204364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7075.575594] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7077.975609] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7079.886623] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7082.040381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7084.481455] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7088.186333] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7105.781192] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7107.692685] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7109.042317] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7110.613685] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7112.261227] Lustre: Failing over lustre-MDT0000 [ 7112.506019] Lustre: server umount lustre-MDT0000 complete [ 7115.233648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7115.238441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7115.248349] Lustre: Skipped 1 previous similar message [ 7115.251735] LustreError: 346462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7115.268643] LustreError: 346462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7116.266860] LustreError: 347190:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7116.292347] LustreError: 347190:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7120.356507] LustreError: 346457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7125.463842] LustreError: 346457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7125.501951] LustreError: 346457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7129.161547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7129.209591] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7129.306802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7129.308900] Lustre: Skipped 1 previous similar message [ 7129.332557] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7130.555902] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7132.125600] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7134.695731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7134.709551] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7134.737590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7138.688927] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7139.861844] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7142.769955] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7144.275810] Lustre: Failing over lustre-MDT0000 [ 7144.479756] Lustre: server umount lustre-MDT0000 complete [ 7144.929162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7144.931591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.946589] Lustre: Skipped 2 previous similar messages [ 7144.957373] LustreError: 349434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7144.967394] LustreError: 349434:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7155.169223] LustreError: 346462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.174670] LustreError: 346462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 7158.814208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7158.874772] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7159.013879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7159.045020] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7160.979508] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7161.304690] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7164.394219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7164.396777] Lustre: Skipped 2 previous similar messages [ 7164.410532] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7164.429890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7168.032772] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7169.597984] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7179.750236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7179.751140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7179.763193] Lustre: Skipped 2 previous similar messages [ 7179.765994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7179.766019] Lustre: Skipped 1 previous similar message [ 7182.272831] Lustre: server umount lustre-MDT0000 complete [ 7184.871225] LustreError: 346459:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7184.921849] LustreError: 346459:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7185.970152] LustreError: 346442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789309251 with bad export cookie 5465153401618591992 [ 7185.974927] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7185.978860] LustreError: 346442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7186.243860] Lustre: server umount lustre-MDT0001 complete [ 7199.833807] Lustre: server umount lustre-OST0000 complete [ 7222.270956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7223.567739] LustreError: 352823:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7223.731192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7232.498954] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7246.341096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7251.910461] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7265.071131] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7278.882686] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7288.621055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7289.079887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7289.082668] Lustre: Skipped 1 previous similar message [ 7293.165508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7296.031745] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7304.727317] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7311.841201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7311.845289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.860762] Lustre: Skipped 2 previous similar messages [ 7311.869878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7311.881062] Lustre: Skipped 2 previous similar messages [ 7312.236319] Lustre: server umount lustre-OST0000 complete [ 7318.502246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7326.307247] Lustre: server umount lustre-MDT0000 complete [ 7329.250685] LustreError: 354669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7329.267425] LustreError: 354669:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7330.291355] LustreError: 353560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789309395 with bad export cookie 5465153401618592741 [ 7330.305121] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7330.317977] LustreError: 353560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7330.960944] Lustre: server umount lustre-MDT0001 complete [ 7342.999736] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 7345.922333] Key type lgssc unregistered [ 7346.270612] LNet: 356357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7346.294830] LNetError: 356357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7346.315697] LNet: Removed LNI 192.168.202.111@tcp [ 7347.264408] Key type .llcrypt unregistered [ 7347.266955] Key type ._llcrypt unregistered [ 7377.547521] Key type ._llcrypt registered [ 7377.551543] Key type .llcrypt registered [ 7377.662810] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 7379.203254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7379.236491] alg: No test for adler32 (adler32-zlib) [ 7380.252286] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 7380.557624] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 7382.232322] Key type lgssc registered [ 7383.645643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7393.040544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7393.057126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7394.748306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7399.944508] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7404.010899] LustreError: 357873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7404.035209] LustreError: 357873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7409.133630] LustreError: 357874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7409.262633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7409.931566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7415.223286] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7423.611992] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7429.449546] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7437.424441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7437.796340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7438.859902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7445.613107] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7455.220616] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7466.753684] Lustre: server umount lustre-MDT0000 complete [ 7469.539955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7469.542111] LustreError: 357870:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7469.547473] Lustre: Skipped 2 previous similar messages [ 7469.570820] LustreError: 357870:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7470.655928] LustreError: 357852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789309535 with bad export cookie 14181643048232113150 [ 7470.672721] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7470.674050] LustreError: 357852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7471.151762] Lustre: server umount lustre-MDT0001 complete [ 7486.509426] Lustre: server umount lustre-OST0000 complete [ 7502.467611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7503.236236] LustreError: 361586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7503.354477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7508.455400] LustreError: 361587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7509.089701] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7519.717378] LustreError: 361586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7519.755822] LustreError: 361586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7520.951643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7521.646935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7527.246307] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7537.179553] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7544.805478] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7553.559824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7553.886452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7561.287337] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7572.206965] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7574.453809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7593.444918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7593.458651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.468180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7594.982377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7594.998956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7595.015611] Lustre: Skipped 1 previous similar message [ 7597.442692] Lustre: server umount lustre-MDT0000 complete [ 7600.104823] LustreError: 361587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7600.132262] LustreError: 361587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7601.957043] LustreError: 362320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789309667 with bad export cookie 14181643048232114263 [ 7601.960485] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7601.968203] LustreError: 362320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7602.514966] Lustre: server umount lustre-MDT0001 complete [ 7618.768703] Lustre: server umount lustre-OST0000 complete [ 7639.707221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7640.390700] LustreError: 365504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7640.467424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7645.938963] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7657.662415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.270786] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7674.136284] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7683.149486] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7691.763390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7692.334550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7692.337255] Lustre: Skipped 1 previous similar message [ 7702.392608] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7710.791534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7713.970373] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7720.936141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7720.945360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7720.954038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7720.974316] Lustre: Skipped 1 previous similar message [ 7721.013019] Lustre: Skipped 3 previous similar messages [ 7726.055414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7726.971476] Lustre: server umount lustre-OST0000 complete [ 7735.266791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7735.272048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.286690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7735.291049] Lustre: Skipped 1 previous similar message [ 7738.000835] Lustre: server umount lustre-MDT0000 complete [ 7741.421018] LustreError: 365501:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7741.449600] LustreError: 365501:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7742.477439] LustreError: 365486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789309807 with bad export cookie 14181643048232115285 [ 7742.484980] LustreError: 365486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7742.485283] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7742.823281] Lustre: server umount lustre-MDT0001 complete [ 7757.542659] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 7761.138635] Key type lgssc unregistered [ 7761.505788] LNet: 369044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7761.510637] LNetError: 369044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7761.542425] LNet: Removed LNI 192.168.202.111@tcp [ 7762.592612] Key type .llcrypt unregistered [ 7762.596600] Key type ._llcrypt unregistered [ 7795.396899] Key type ._llcrypt registered [ 7795.403474] Key type .llcrypt registered [ 7795.605086] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 7797.487480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7797.529567] alg: No test for adler32 (adler32-zlib) [ 7798.886422] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 7799.322387] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 7801.080212] Key type lgssc registered [ 7802.450459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7811.070366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7811.107152] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7811.121376] LustreError: 370486:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7811.162657] LustreError: 370486:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7811.178205] LustreError: 370486:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7811.192862] LustreError: 370486:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7811.201590] LustreError: 370486:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7819.915121] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7819.924534] LustreError: 370883:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7819.952555] LustreError: 370883:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7819.967536] LustreError: 370883:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7819.979354] LustreError: 370883:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7819.991389] LustreError: 370883:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7845.357830] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 7849.546355] Key type lgssc unregistered [ 7849.931090] LNet: 371886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7849.952386] LNetError: 371886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7849.975817] LNet: Removed LNI 192.168.202.111@tcp [ 7850.995525] Key type .llcrypt unregistered [ 7850.997081] Key type ._llcrypt unregistered [ 7872.263478] Key type ._llcrypt registered [ 7872.268090] Key type .llcrypt registered [ 7872.363545] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 7890.594836] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 7891.708450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7891.788819] alg: No test for adler32 (adler32-zlib) [ 7893.125516] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 7893.401753] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 7895.112218] Key type lgssc registered [ 7896.406634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7940.112118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7940.147707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7941.517206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7941.549426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7941.602641] Lustre: lustre-MDT0000: new disk, initializing [ 7941.663391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7941.705663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7946.175820] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7958.262899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7958.448241] Lustre: 375042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7958.509505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7958.514090] Lustre: Skipped 1 previous similar message [ 7958.603339] Lustre: lustre-MDT0001: new disk, initializing [ 7958.700940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7958.758944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7958.777892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7964.189243] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7975.348983] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7982.807125] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7991.489769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7991.988398] Lustre: lustre-OST0000: new disk, initializing [ 7991.991837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7992.007031] Lustre: 377012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7992.175140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7999.580628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7999.590465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7999.734306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8000.271697] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8011.819247] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8017.879288] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8018.134392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8023.947757] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8024.319355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8030.692597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8030.697916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8030.703399] LustreError: Skipped 1 previous similar message [ 8030.712659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8030.725271] Lustre: Skipped 1 previous similar message [ 8030.750666] Lustre: Skipped 1 previous similar message [ 8032.898093] Lustre: server umount lustre-OST0000 complete [ 8040.936946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8040.939525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8040.977641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8043.486785] Lustre: server umount lustre-MDT0000 complete [ 8048.199744] LustreError: 375033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789310113 with bad export cookie 17345333914250035902 [ 8048.233750] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8048.931437] Lustre: server umount lustre-MDT0001 complete [ 8061.063634] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8062.802105] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:35:26 (1789310126) [ 8070.454513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8071.027840] LustreError: 379474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8071.057693] LustreError: 379474:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8071.127177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8076.258930] LustreError: 379475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8076.829929] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8081.380409] LustreError: 379474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8086.500551] LustreError: 379475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8086.536617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8086.864808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8092.171041] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8101.412593] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8108.328931] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8116.179272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8116.554249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8123.727089] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8133.273972] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8140.468030] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8140.735858] Lustre: lustre-OST0001: new disk, initializing [ 8140.742374] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8140.753180] Lustre: 382313:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8140.868592] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8147.023958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8147.039374] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8147.151826] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8148.111737] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8158.915709] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8166.896303] Lustre: lustre-OST0001: Not available for connect from 192.168.202.11@tcp (stopping) [ 8166.901557] Lustre: Skipped 1 previous similar message [ 8167.926104] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8167.932902] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8167.968563] Lustre: Skipped 1 previous similar message [ 8169.973314] Lustre: server umount lustre-OST0001 complete [ 8172.000833] LustreError: 381296:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8183.275063] LustreError: 381299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8183.308397] LustreError: 381299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 8188.392433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8188.401432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8188.424842] Lustre: Skipped 2 previous similar messages [ 8188.431710] Lustre: Skipped 2 previous similar messages [ 8193.509484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8193.516394] Lustre: Skipped 2 previous similar messages [ 8202.210113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8202.485985] Lustre: server umount lustre-OST0000 complete [ 8208.866264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.886750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8208.893534] Lustre: Skipped 3 previous similar messages [ 8213.452909] Lustre: server umount lustre-MDT0000 complete [ 8213.996232] LustreError: 379470:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8214.018304] LustreError: 379470:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 8217.458154] LustreError: 379456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789310282 with bad export cookie 17345333914250036784 [ 8217.478720] LustreError: 379456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8217.483651] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8224.041698] Lustre: server umount lustre-MDT0001 complete [ 8237.235194] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 8240.785992] Key type lgssc unregistered [ 8241.193842] LNet: 384302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8241.204707] LNetError: 384302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8241.229597] LNet: Removed LNI 192.168.202.111@tcp [ 8242.285941] Key type .llcrypt unregistered [ 8242.288340] Key type ._llcrypt unregistered [ 8296.995738] Key type ._llcrypt registered [ 8296.999741] Key type .llcrypt registered [ 8297.138972] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 8298.172815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8298.204907] alg: No test for adler32 (adler32-zlib) [ 8299.623822] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 8299.873992] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 8301.547142] Key type lgssc registered [ 8302.556247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8310.941327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8310.953521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8312.324543] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8312.350711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8312.588879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8317.307485] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8327.088386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8327.326803] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8327.371672] Lustre: 387098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8327.613370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8332.487651] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8341.278493] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8349.263621] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8353.255107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8353.272407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.303909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8354.788422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8354.806298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8358.907166] Lustre: server umount lustre-MDT0000 complete [ 8364.215781] LustreError: 387089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789310429 with bad export cookie 8971517525577531374 [ 8364.231751] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8365.288799] Lustre: server umount lustre-MDT0001 complete [ 8378.157802] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 10:40:41 (1789310441) [ 8387.152751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8388.066769] LustreError: 389613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8388.093022] LustreError: 389613:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8388.123899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8393.101479] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8393.197091] LustreError: 389614:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8398.307520] LustreError: 389613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8403.425221] LustreError: 389614:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8403.538155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8404.006242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8408.882888] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8417.872713] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8424.745539] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8432.190824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8432.375761] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8432.546073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8439.469572] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8448.967592] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8458.737083] Lustre: Permanently deactivating lustre-OST0000 [ 8458.748979] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8464.932511] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8466.760189] LustreError: 392557:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8466.778191] LustreError: 392557:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8466.783854] LustreError: 392557:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8470.695691] LustreError: 392606:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8470.705950] LustreError: 392606:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8479.420267] Lustre: server umount lustre-OST0000 complete [ 8482.148695] LustreError: 392855:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8482.152313] LustreError: 392855:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8485.870213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8485.881258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8485.889399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8485.891988] Lustre: Skipped 1 previous similar message [ 8492.005610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8492.015548] Lustre: Skipped 3 previous similar messages [ 8494.246713] Lustre: server umount lustre-MDT0000 complete [ 8496.034294] LustreError: 389608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8496.063428] LustreError: 389608:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8498.225300] LustreError: 389593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789310563 with bad export cookie 8971517525577531955 [ 8498.230754] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8498.237730] LustreError: 389593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8508.158087] Lustre: server umount lustre-MDT0001 complete [ 8518.371756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8523.064715] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8526.178617] LustreError: 394015:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8533.315421] Lustre: 394354:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.111@tcp [ 8536.012360] Lustre: server umount MGS complete [ 8549.841088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8550.351539] LustreError: 395079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8550.371385] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8550.375351] Lustre: Skipped 1 previous similar message [ 8550.422098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8555.190361] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8564.610158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8564.915470] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8570.062678] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8578.686644] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8584.274342] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8586.111266] Lustre: Permanently reactivating lustre-OST0000 [ 8586.119661] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8586.122635] Lustre: Skipped 2 previous similar messages [ 8613.857398] LustreError: 396860:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8621.029935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8621.364870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8621.371648] Lustre: Skipped 1 previous similar message [ 8628.329258] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8635.630425] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8642.414606] Lustre: server umount lustre-OST0000 complete [ 8647.137655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8647.148356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8647.160437] Lustre: Skipped 1 previous similar message [ 8647.164867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8655.332473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8655.350916] Lustre: Skipped 3 previous similar messages [ 8657.497361] Lustre: server umount lustre-MDT0000 complete [ 8660.458110] LustreError: 395075:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8660.459185] Lustre: 386418:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789310709/real 1789310709] req@000000002c31859d x1876227905392000/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789310725 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8660.492000] LustreError: 395075:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8660.555574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8660.569607] Lustre: Skipped 1 previous similar message [ 8661.365467] LustreError: 395061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789310726 with bad export cookie 8971517525577533005 [ 8661.366218] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8661.375385] LustreError: 395061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8661.806251] Lustre: server umount lustre-MDT0001 complete [ 8674.286470] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 8677.541026] Key type lgssc unregistered [ 8677.880092] LNet: 399493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8677.891340] LNetError: 399493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8677.909924] LNet: Removed LNI 192.168.202.111@tcp [ 8679.175536] Key type .llcrypt unregistered [ 8679.178097] Key type ._llcrypt unregistered [ 8697.768990] Key type ._llcrypt registered [ 8697.770752] Key type .llcrypt registered [ 8697.866153] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 8713.197149] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 8714.889494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8714.933430] alg: No test for adler32 (adler32-zlib) [ 8715.996815] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 8716.310323] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 8717.968203] Key type lgssc registered [ 8719.397136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8755.354442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8755.382725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8756.781480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8756.828221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8756.992518] Lustre: lustre-MDT0000: new disk, initializing [ 8757.126111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8757.159787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8761.557272] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8773.716645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8773.842928] Lustre: 402649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8773.918679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8773.921365] Lustre: Skipped 1 previous similar message [ 8773.979377] Lustre: lustre-MDT0001: new disk, initializing [ 8774.053631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8774.080366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8774.090382] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8778.954458] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8788.918955] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8795.024876] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8801.040924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8801.368335] Lustre: lustre-OST0000: new disk, initializing [ 8801.372647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8801.379837] Lustre: 404617:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8801.471913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8808.264760] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8809.117349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8809.123317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8809.238876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8817.419875] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8822.768219] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8823.069435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8829.119693] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8829.409374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8835.049294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.063236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8835.065706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8835.091161] Lustre: Skipped 1 previous similar message [ 8838.611430] Lustre: server umount lustre-OST0000 complete [ 8845.283362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8845.304391] Lustre: Skipped 1 previous similar message [ 8845.312784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8849.170641] Lustre: server umount lustre-MDT0000 complete [ 8852.654737] LustreError: 402640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789310917 with bad export cookie 17763803382839733727 [ 8852.660184] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8852.663525] LustreError: 402640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8853.058852] Lustre: server umount lustre-MDT0001 complete [ 8864.071089] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8866.209553] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:48:49 (1789310929) [ 8874.612332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8875.035111] LustreError: 407079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8875.055980] LustreError: 407079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8875.123108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8880.498987] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8880.620981] LustreError: 407080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8885.754354] LustreError: 407079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8890.849050] LustreError: 407080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8891.542784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8892.182687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8897.897181] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8907.700219] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8914.531260] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8922.685026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8923.065736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8930.478551] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8939.217590] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8941.823524] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8949.004274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8955.881546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8955.884143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8955.905522] Lustre: Skipped 1 previous similar message [ 8955.906691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8955.924564] Lustre: Skipped 2 previous similar messages [ 8960.994834] LustreError: 409717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8961.087852] Lustre: server umount lustre-OST0000 complete [ 8966.114160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8966.116887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8966.123056] Lustre: Skipped 2 previous similar messages [ 8966.146473] Lustre: Skipped 1 previous similar message [ 8971.239340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8971.262586] Lustre: Skipped 1 previous similar message [ 8971.704299] Lustre: server umount lustre-MDT0000 complete [ 8976.356856] LustreError: 407079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8976.382777] LustreError: 407079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8976.485695] LustreError: 407061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789311041 with bad export cookie 17763803382839734609 [ 8976.500110] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8976.498538] LustreError: 407061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8977.104249] Lustre: server umount lustre-MDT0001 complete [ 8989.775245] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 8993.196794] Key type lgssc unregistered [ 8993.650749] LNet: 410808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8993.665254] LNetError: 410808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8993.704801] LNet: Removed LNI 192.168.202.111@tcp [ 8994.832482] Key type .llcrypt unregistered [ 8994.838765] Key type ._llcrypt unregistered [ 9025.645339] Key type ._llcrypt registered [ 9025.649892] Key type .llcrypt registered [ 9025.809296] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 9026.178648] Key type .llcrypt unregistered [ 9026.183868] Key type ._llcrypt unregistered [ 9045.077415] Key type ._llcrypt registered [ 9045.079476] Key type .llcrypt registered [ 9045.185257] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 9046.547817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9046.633775] alg: No test for adler32 (adler32-zlib) [ 9047.814308] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 9048.174194] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 9049.922599] Key type lgssc registered [ 9051.318238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9059.930131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9059.946672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9061.543512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9066.969537] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9071.074318] LustreError: 413014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9071.092896] LustreError: 413014:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9076.195181] LustreError: 413015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9077.845801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9078.253470] LustreError: 413014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9078.275672] LustreError: 413014:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9078.303861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9083.365440] LustreError: 413010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9083.395754] LustreError: 413010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9085.807379] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9086.470855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9091.556878] LustreError: 413015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9091.578267] LustreError: 413015:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9094.960136] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9101.795857] LustreError: 413011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9101.823257] LustreError: 413011:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9102.516782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9102.891609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9108.474956] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9118.194123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9118.194738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9118.216763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9118.222708] Lustre: Skipped 1 previous similar message [ 9123.309443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9123.326308] Lustre: Skipped 2 previous similar messages [ 9124.389418] Lustre: server umount lustre-OST0000 complete [ 9133.545378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9133.546519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9133.556140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9133.571993] Lustre: Skipped 1 previous similar message [ 9135.058757] Lustre: server umount lustre-MDT0000 complete [ 9138.660103] LustreError: 414754:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9138.684889] LustreError: 414754:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9138.697201] LustreError: 414760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789311203 with bad export cookie 10243607466824310595 [ 9138.706505] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9138.709796] LustreError: 414760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9139.148654] Lustre: server umount lustre-MDT0001 complete [ 9152.738815] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 9155.629736] Key type lgssc unregistered [ 9155.961707] LNet: 416164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9155.976298] LNetError: 416164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9156.005891] LNet: Removed LNI 192.168.202.111@tcp [ 9157.013080] Key type .llcrypt unregistered [ 9157.014638] Key type ._llcrypt unregistered [ 9186.699422] Key type ._llcrypt registered [ 9186.702251] Key type .llcrypt registered [ 9186.896694] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 9188.343345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9188.672618] alg: No test for adler32 (adler32-zlib) [ 9189.845164] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 9190.120222] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 9191.896505] Key type lgssc registered [ 9193.197403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9202.352634] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9202.391510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9218.145053] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9223.264578] LustreError: 417554:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.111@tcp: failed processing log, type 4: rc = -110 [ 9249.952197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9257.459219] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9266.117983] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9274.309485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9275.108603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9275.195646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9280.484957] LustreError: 417579:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9280.512530] LustreError: 417579:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9281.088636] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9284.581079] LustreError: 417580:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9284.613394] LustreError: 417580:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9289.710369] LustreError: 418590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9289.737524] LustreError: 418590:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9291.064738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9291.610167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9296.849990] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9305.529812] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9311.899650] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9321.443064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9321.454796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9321.504949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9322.465954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9322.478954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9322.480870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9322.508237] Lustre: Skipped 1 previous similar message [ 9326.597666] Lustre: server umount lustre-OST0000 complete [ 9332.705893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9332.711437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9332.717443] Lustre: Skipped 1 previous similar message [ 9332.725402] Lustre: Skipped 1 previous similar message [ 9333.241726] Lustre: server umount lustre-MDT0000 complete [ 9337.600761] LustreError: 417561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789311402 with bad export cookie 7668953352130288922 [ 9337.614456] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9337.889860] Lustre: server umount lustre-MDT0001 complete [ 9350.843048] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 9354.017602] Key type lgssc unregistered [ 9354.285300] LNet: 421125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9354.289887] LNetError: 421125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9355.312452] LNet: Removed LNI 192.168.202.111@tcp [ 9356.271632] Key type .llcrypt unregistered [ 9356.274674] Key type ._llcrypt unregistered [ 9385.325552] Key type ._llcrypt registered [ 9385.327554] Key type .llcrypt registered [ 9385.458763] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 9387.042610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9387.092506] alg: No test for adler32 (adler32-zlib) [ 9388.415936] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 9388.794410] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 9390.616485] Key type lgssc registered [ 9391.937810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9400.396056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9400.435881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9402.015809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9407.110174] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9412.588488] LustreError: 422549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9412.618039] LustreError: 422549:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9416.883295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9417.254355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9422.950539] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9431.735486] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9438.701619] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9447.364578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9447.827150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9455.173534] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9463.867728] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9472.389569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9481.252600] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.11@tcp (stopping) [ 9482.742050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9482.761985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9482.779456] Lustre: Skipped 2 previous similar messages [ 9482.795106] Lustre: Skipped 2 previous similar messages [ 9486.300582] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.11@tcp (stopping) [ 9486.906836] Lustre: server umount lustre-MDT0000 complete [ 9487.845245] LustreError: 422549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9487.864820] LustreError: 422549:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9490.231479] LustreError: 422530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.11@tcp arrived at 1789311555 with bad export cookie 2471382605665056232 [ 9492.240330] LustreError: 422528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789311557 with bad export cookie 2471382605665055567 [ 9492.251385] LustreError: 422528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9492.257710] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9492.962373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9492.968558] LustreError: 422550:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9492.970510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9492.974775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9493.010838] LustreError: 422550:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9498.081633] LustreError: 423985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9498.082776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9498.094995] LustreError: 423985:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9498.648148] Lustre: server umount lustre-OST0000 complete [ 9502.179261] LustreError: 422550:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9502.198376] LustreError: 422550:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9507.727659] Lustre: server umount lustre-MDT0001 complete [ 9521.794402] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 9525.310578] Key type lgssc unregistered [ 9525.833721] LNet: 426255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9525.842332] LNetError: 426255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9525.880414] LNet: Removed LNI 192.168.202.111@tcp [ 9526.778682] Key type .llcrypt unregistered [ 9526.785212] Key type ._llcrypt unregistered [ 9555.762238] Key type ._llcrypt registered [ 9555.763565] Key type .llcrypt registered [ 9556.007453] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 9557.652379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9557.711166] alg: No test for adler32 (adler32-zlib) [ 9558.917498] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 9559.151584] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 9560.920530] Key type lgssc registered [ 9562.555410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9571.608565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9571.624573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9573.275548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9578.577711] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9583.593711] LustreError: 427677:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9583.619602] LustreError: 427677:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9588.664184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9588.709236] LustreError: 427676:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9589.179935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9594.884465] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9604.091582] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9611.179236] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9618.631843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9618.997950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9625.415679] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9633.083851] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9643.620213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9648.611650] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9648.628463] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9648.631087] Lustre: Skipped 1 previous similar message [ 9650.172330] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.11@tcp (stopping) [ 9653.765896] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9653.775916] Lustre: Skipped 1 previous similar message [ 9655.030126] Lustre: server umount lustre-MDT0001 complete [ 9655.256848] LustreError: 428784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9658.854888] LustreError: 427676:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9658.872016] LustreError: 427676:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9663.969565] LustreError: 428628:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9663.980150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9664.021603] Lustre: Skipped 1 previous similar message [ 9664.030751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9668.311393] Lustre: server umount lustre-OST0000 complete [ 9671.842974] Lustre: server umount lustre-MDT0000 complete [ 9688.345400] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 9691.099303] Key type lgssc unregistered [ 9691.516391] LNet: 431397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9691.528748] LNetError: 431397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9691.557787] LNet: Removed LNI 192.168.202.111@tcp [ 9692.260516] Key type .llcrypt unregistered [ 9692.262648] Key type ._llcrypt unregistered [ 9723.766288] Key type ._llcrypt registered [ 9723.768557] Key type .llcrypt registered [ 9723.901652] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 9741.739922] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 9743.267677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9743.307352] alg: No test for adler32 (adler32-zlib) [ 9744.442820] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 9744.679052] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 9746.384285] Key type lgssc registered [ 9747.578137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9794.286092] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 9812.710073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9812.734797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9814.215319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9814.260454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9814.411866] Lustre: lustre-MDT0000: new disk, initializing [ 9814.638888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9814.673972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9820.836642] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9837.408911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9837.814451] Lustre: lustre-OST0000: new disk, initializing [ 9837.821690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9837.827114] Lustre: Skipped 1 previous similar message [ 9837.834337] Lustre: 436431:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9837.949634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9845.334117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9845.345311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9845.383121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9845.703749] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9857.993333] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9876.914404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9877.094989] Lustre: 435412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9877.239991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9877.354200] Lustre: lustre-MDT0001: new disk, initializing [ 9877.478492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9877.523497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 9877.543900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 9883.126792] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9895.635623] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9896.003813] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9900.139903] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9900.245980] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9910.249731] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9910.251181] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9910.272789] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9910.282545] Lustre: Skipped 2 previous similar messages [ 9911.750499] Lustre: server umount lustre-MDT0001 complete [ 9920.512141] LustreError: 437772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9920.572745] LustreError: 437772:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9925.626677] LustreError: 435420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9925.652405] LustreError: 435420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9930.725657] LustreError: 437772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9930.728126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9930.767507] Lustre: Skipped 1 previous similar message [ 9930.791511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9935.842603] LustreError: 435425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9935.842952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9936.742595] Lustre: server umount lustre-OST0000 complete [ 9940.981403] LustreError: 437772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9947.739317] Lustre: server umount lustre-MDT0000 complete [ 9963.923665] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 9967.692813] Key type lgssc unregistered [ 9968.022387] LNet: 440181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9968.037355] LNetError: 440181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9968.069928] LNet: Removed LNI 192.168.202.111@tcp [ 9969.161517] Key type .llcrypt unregistered [ 9969.163688] Key type ._llcrypt unregistered [10014.268797] Key type ._llcrypt registered [10014.271411] Key type .llcrypt registered [10014.419260] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [10014.746336] Key type .llcrypt unregistered [10014.749608] Key type ._llcrypt unregistered [10034.567088] Key type ._llcrypt registered [10034.573257] Key type .llcrypt registered [10034.791039] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [10036.502531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10036.812060] alg: No test for adler32 (adler32-zlib) [10038.055849] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [10038.438729] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [10040.189165] Key type lgssc registered [10041.793685] Lustre: Echo OBD driver; http://www.lustre.org/ [10073.109272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10073.142814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10074.432269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10074.457837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10074.522781] Lustre: lustre-MDT0000: new disk, initializing [10074.606576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10074.632583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10079.738949] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10091.185331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.307559] Lustre: 443554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10091.361598] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10091.364262] Lustre: Skipped 1 previous similar message [10091.439402] Lustre: lustre-MDT0001: new disk, initializing [10091.554197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10091.618548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10091.635502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10096.956800] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10107.921597] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10114.494644] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10122.336024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10122.694537] Lustre: lustre-OST0000: new disk, initializing [10122.699694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10122.716703] Lustre: 445522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10122.840764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10129.484467] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10130.512407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10130.529886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10130.711619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10140.206803] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10151.407876] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10151.410676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10151.442797] Lustre: Skipped 2 previous similar messages [10151.467947] Lustre: Skipped 2 previous similar messages [10155.337974] Lustre: server umount lustre-MDT0000 complete [10159.591286] LustreError: 443549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789312224 with bad export cookie 5103308605759247891 [10159.619399] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10159.620467] LustreError: 443549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10160.172490] Lustre: server umount lustre-MDT0001 complete [10174.352342] Lustre: server umount lustre-OST0000 complete [10186.504106] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:10:49 (1789312249) [10195.395466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10195.850348] LustreError: 447552:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10195.867600] LustreError: 447552:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10195.956177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10201.071721] LustreError: 447553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10201.201539] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10206.178259] LustreError: 447552:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10210.055637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10210.536912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10214.878876] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10222.948569] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10229.545471] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10250.734725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10250.902768] Lustre: Found index 0 for lustre-OST0000, updating log [10251.124222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10258.656208] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10267.510566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10267.634320] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10272.738440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10272.743948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10272.751249] Lustre: Skipped 1 previous similar message [10272.765879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 1.2.3.4@tcp failed: rc = -107 [10273.834051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10273.838277] Lustre: Skipped 1 previous similar message [10274.666191] Lustre: server umount lustre-OST0000 complete [10290.147946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10290.171286] Lustre: Skipped 1 previous similar message [10290.192430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10290.198339] Lustre: Skipped 2 previous similar messages [10295.268929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10295.281115] Lustre: Skipped 1 previous similar message [10296.556754] Lustre: server umount lustre-MDT0000 complete [10299.695120] LustreError: 447534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789312364 with bad export cookie 5103308605759249284 [10299.697396] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10299.702984] LustreError: 447534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10300.048434] Lustre: server umount lustre-MDT0001 complete [10308.279712] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:12:51 (1789312371) [10315.936433] LNet: 1 local NIs in recovery (showing 1): 192.168.202.111@tcp [10918.704314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10919.110805] LustreError: 453960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10919.176729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10922.741919] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10924.514848] LustreError: 453961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10929.248639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10929.472940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10933.466650] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10938.996831] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10943.755490] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10962.090907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10962.296922] Lustre: Found index 0 for lustre-OST0000, updating log [10962.542256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10967.242136] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10968.688630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [10972.816232] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10976.293144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10976.299304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10980.341273] Lustre: server umount lustre-OST0000 complete [10980.514879] LNet: 1 local NIs in recovery (showing 1): 192.168.202.111@tcp [10984.225800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10984.230215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10984.241963] Lustre: Skipped 1 previous similar message [10986.339131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10986.347284] Lustre: Skipped 4 previous similar messages [10989.557043] Lustre: server umount lustre-MDT0000 complete [10990.440882] LustreError: 453956:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10990.460764] LustreError: 453956:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10992.484058] LustreError: 455252:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10992.497022] LustreError: 455252:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10992.765425] LustreError: 453941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789313057 with bad export cookie 5103308605759250369 [10992.772876] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10992.780176] LustreError: 453941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10993.057679] Lustre: server umount lustre-MDT0001 complete [11018.139455] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [11025.188105] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [11052.304092] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 11:25:15 (1789313115) [11063.545763] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [11065.232502] Key type lgssc unregistered [11065.419428] LNet: 460569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11065.422972] LNetError: 460569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11179.111715] LNet: Removed LNI 192.168.202.111@tcp [11179.585590] Key type .llcrypt unregistered [11179.591578] Key type ._llcrypt unregistered [11189.832502] Key type ._llcrypt registered [11189.837448] Key type .llcrypt registered [11189.904378] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [11190.540666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11190.559119] alg: No test for adler32 (adler32-zlib) [11191.518191] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [11191.673284] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [11193.296193] Key type lgssc registered [11193.911574] Lustre: Echo OBD driver; http://www.lustre.org/ [11196.331813] LNet: Added LNI 192.168.202.111@tcp100 [8/256/0/180] [11196.853346] LNet: Added LNI 192.168.202.111@tcp101 [8/256/0/180] [11197.396826] LNet: Added LNI 192.168.202.111@tcp102 [8/256/0/180] [11197.913336] LNet: Added LNI 192.168.202.111@tcp103 [8/256/0/180] [11198.439579] LNet: Added LNI 192.168.202.111@tcp104 [8/256/0/180] [11198.932699] LNet: Added LNI 192.168.202.111@tcp105 [8/256/0/180] [11199.468969] LNet: Added LNI 192.168.202.111@tcp106 [8/256/0/180] [11199.978490] LNet: Added LNI 192.168.202.111@tcp107 [8/256/0/180] [11200.552308] LNet: Added LNI 192.168.202.111@tcp108 [8/256/0/180] [11201.161785] LNet: Added LNI 192.168.202.111@tcp109 [8/256/0/180] [11201.689674] LNet: Added LNI 192.168.202.111@tcp110 [8/256/0/180] [11202.269176] LNet: Added LNI 192.168.202.111@tcp111 [8/256/0/180] [11202.910273] LNet: Added LNI 192.168.202.111@tcp112 [8/256/0/180] [11203.476881] LNet: Added LNI 192.168.202.111@tcp113 [8/256/0/180] [11204.023848] LNet: Added LNI 192.168.202.111@tcp114 [8/256/0/180] [11204.575984] LNet: Added LNI 192.168.202.111@tcp115 [8/256/0/180] [11205.122582] LNet: Added LNI 192.168.202.111@tcp116 [8/256/0/180] [11205.684965] LNet: Added LNI 192.168.202.111@tcp117 [8/256/0/180] [11206.273760] LNet: Added LNI 192.168.202.111@tcp118 [8/256/0/180] [11206.828871] LNet: Added LNI 192.168.202.111@tcp119 [8/256/0/180] [11207.382391] LNet: Added LNI 192.168.202.111@tcp120 [8/256/0/180] [11207.992603] LNet: Added LNI 192.168.202.111@tcp121 [8/256/0/180] [11208.507854] LNet: Added LNI 192.168.202.111@tcp122 [8/256/0/180] [11209.058280] LNet: Added LNI 192.168.202.111@tcp123 [8/256/0/180] [11209.620325] LNet: Added LNI 192.168.202.111@tcp124 [8/256/0/180] [11210.200694] LNet: Added LNI 192.168.202.111@tcp125 [8/256/0/180] [11210.860111] LNet: Added LNI 192.168.202.111@tcp126 [8/256/0/180] [11211.462423] LNet: Added LNI 192.168.202.111@tcp127 [8/256/0/180] [11212.038502] LNet: Added LNI 192.168.202.111@tcp128 [8/256/0/180] [11212.619376] LNet: Added LNI 192.168.202.111@tcp129 [8/256/0/180] [11213.140354] LNet: Added LNI 192.168.202.111@tcp130 [8/256/0/180] [11213.606563] LNet: Added LNI 192.168.202.111@tcp131 [8/256/0/180] [11214.121327] LNet: Added LNI 192.168.202.111@tcp132 [8/256/0/180] [11214.614791] LNet: Added LNI 192.168.202.111@tcp133 [8/256/0/180] [11215.144720] LNet: Added LNI 192.168.202.111@tcp134 [8/256/0/180] [11215.627817] LNet: Added LNI 192.168.202.111@tcp135 [8/256/0/180] [11219.290797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11219.300966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11220.354754] Lustre: 463575:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.111@tcp: can't add NID 192.168.202.111@tcp130: rc = -75 [11220.398383] LNetError: 463607:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.111 [11220.407635] LNetError: Protocol error connecting to 192.168.202.111@tcp122 at host 192.168.202.111:988: is it running a compatible version of Lustre? [11220.412808] LNetError: Skipped 4 previous similar messages [11220.461428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11220.473951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11220.532606] Lustre: lustre-MDT0000: new disk, initializing [11220.564959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11220.584613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11221.409127] LNet: 3 local NIs in recovery (showing 3): 192.168.202.111@tcp128, 192.168.202.111@tcp129, 192.168.202.111@tcp122 [11221.410045] LNetError: 463601:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.111 [11221.422104] LNetError: 463601:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [11221.425449] LNetError: Protocol error connecting to 192.168.202.111@tcp110 at host 192.168.202.111:988: is it running a compatible version of Lustre? [11221.432539] LNetError: Skipped 3 previous similar messages [11222.798904] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11229.031351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11229.110520] Lustre: 463591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11229.146795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11229.149053] Lustre: Skipped 1 previous similar message [11229.189939] Lustre: lustre-MDT0001: new disk, initializing [11229.210633] Lustre: 464396:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.111@tcp: can't add NID 192.168.202.111@tcp130: rc = -75 [11229.214581] Lustre: 464396:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11229.280695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11229.305848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11229.311302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11231.478038] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11233.064196] Lustre: 464761:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.111@tcp: can't add NID 192.168.202.111@tcp130: rc = -75 [11233.070075] Lustre: 464761:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11237.355134] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11241.005079] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11245.025528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11245.187090] Lustre: lustre-OST0000: new disk, initializing [11245.188808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11245.193993] Lustre: 465568:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11245.240671] Lustre: 465573:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.111@tcp: can't add NID 192.168.202.111@tcp130: rc = -75 [11245.243786] Lustre: 465573:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [11245.270840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11248.204594] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11250.017993] LNetError: 463607:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.111 [11250.023929] LNetError: 463607:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [11250.026801] LNetError: Protocol error connecting to 192.168.202.111@tcp118 at host 192.168.202.111:988: is it running a compatible version of Lustre? [11251.212330] Lustre: 466094:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.111@tcp: can't add NID 192.168.202.111@tcp130: rc = -75 [11251.220130] Lustre: 466094:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11251.262555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11251.268676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11251.376640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11253.089355] LNetError: 461032:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.111 [11253.098640] LNetError: Protocol error connecting to 192.168.202.111@tcp124 at host 192.168.202.111:988: is it running a compatible version of Lustre? [11253.723953] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11258.209368] LNetError: 463610:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.111 [11258.215056] LNetError: 463610:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [11258.218092] LNetError: Protocol error connecting to 192.168.202.111@tcp125 at host 192.168.202.111:988: is it running a compatible version of Lustre? [11258.222661] LNetError: Skipped 2 previous similar messages [11261.924529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11261.925689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11261.931887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11261.940639] Lustre: Skipped 1 previous similar message [11263.617246] Lustre: server umount lustre-OST0000 complete [11266.404189] LNetError: 461031:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.111 [11266.410433] LNetError: 461031:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [11266.417623] LNetError: Protocol error connecting to 192.168.202.111@tcp124 at host 192.168.202.111:988: is it running a compatible version of Lustre? [11266.422584] LNetError: Skipped 3 previous similar messages [11267.041618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11267.046089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11267.048716] Lustre: Skipped 1 previous similar message [11267.054095] Lustre: Skipped 2 previous similar messages [11272.163256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11272.167060] Lustre: Skipped 1 previous similar message [11277.281951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11277.284952] Lustre: Skipped 1 previous similar message [11280.352178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11280.455392] Lustre: server umount lustre-MDT0000 complete [11282.273310] LustreError: 463584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789313347 with bad export cookie 5950438012378731783 [11282.277412] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11282.279978] LustreError: 463584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11282.414712] Lustre: server umount lustre-MDT0001 complete [11282.786527] LNetError: 463610:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.111 [11282.790538] LNetError: 463610:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [11282.793460] LNetError: Protocol error connecting to 192.168.202.111@tcp122 at host 192.168.202.111:988: is it running a compatible version of Lustre? [11282.797360] LNetError: Skipped 8 previous similar messages [11287.310631] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [11288.832550] Key type lgssc unregistered [11289.064453] LNet: 467478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11289.069974] LNetError: 467478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11289.075605] LNet: Removed LNI 192.168.202.111@tcp [11290.080766] LNet: Removed LNI 192.168.202.111@tcp100 [11290.085489] LNetError: 467478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11290.090027] LNetError: 467478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11290.093687] LNet: Removed LNI 192.168.202.111@tcp101 [11291.104170] LNet: Removed LNI 192.168.202.111@tcp102 [11291.106862] LNetError: 467478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11291.113138] LNetError: 467478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11291.117796] LNet: Removed LNI 192.168.202.111@tcp103 [11292.128193] LNet: Removed LNI 192.168.202.111@tcp104 [11293.152240] LNet: Removed LNI 192.168.202.111@tcp105 [11293.157307] LNetError: 467478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11293.165773] LNetError: 467478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11293.169444] LNet: Removed LNI 192.168.202.111@tcp106 [11294.176172] LNet: Removed LNI 192.168.202.111@tcp107 [11295.200139] LNet: Removed LNI 192.168.202.111@tcp108 [11296.224155] LNet: Removed LNI 192.168.202.111@tcp109 [11297.248146] LNet: Removed LNI 192.168.202.111@tcp110 [11297.250425] LNetError: 467478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11297.258762] LNetError: 467478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11297.268085] LNet: Removed LNI 192.168.202.111@tcp111 [11298.272144] LNet: Removed LNI 192.168.202.111@tcp112 [11299.298131] LNet: Removed LNI 192.168.202.111@tcp113 [11300.320377] LNet: Removed LNI 192.168.202.111@tcp114 [11301.345244] LNet: Removed LNI 192.168.202.111@tcp115 [11302.368304] LNet: Removed LNI 192.168.202.111@tcp116 [11303.392141] LNet: Removed LNI 192.168.202.111@tcp117 [11304.416140] LNet: Removed LNI 192.168.202.111@tcp118 [11305.440130] LNet: Removed LNI 192.168.202.111@tcp119 [11305.443485] LNetError: 467478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11305.448585] LNetError: 467478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11305.453203] LNet: Removed LNI 192.168.202.111@tcp120 [11306.464280] LNet: Removed LNI 192.168.202.111@tcp121 [11307.488349] LNet: Removed LNI 192.168.202.111@tcp122 [11308.512133] LNet: Removed LNI 192.168.202.111@tcp123 [11309.536356] LNet: Removed LNI 192.168.202.111@tcp124 [11310.560154] LNet: Removed LNI 192.168.202.111@tcp125 [11311.584181] LNet: Removed LNI 192.168.202.111@tcp126 [11312.608373] LNet: Removed LNI 192.168.202.111@tcp127 [11313.632142] LNet: Removed LNI 192.168.202.111@tcp128 [11314.656225] LNet: Removed LNI 192.168.202.111@tcp129 [11314.659974] LNet: Removed LNI 192.168.202.111@tcp130 [11314.665641] LNet: Removed LNI 192.168.202.111@tcp131 [11314.671298] LNet: Removed LNI 192.168.202.111@tcp132 [11314.675882] LNet: Removed LNI 192.168.202.111@tcp133 [11314.679894] LNet: Removed LNI 192.168.202.111@tcp134 [11314.691794] LNet: Removed LNI 192.168.202.111@tcp135 [11315.203634] Key type .llcrypt unregistered [11315.205930] Key type ._llcrypt unregistered [11328.236217] Key type ._llcrypt registered [11328.238411] Key type .llcrypt registered [11328.314865] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [11338.687789] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [11339.562693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11339.571590] alg: No test for adler32 (adler32-zlib) [11340.546720] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [11340.756340] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [11342.376591] Key type lgssc registered [11343.128343] Lustre: Echo OBD driver; http://www.lustre.org/ [11362.551213] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 11:30:26 (1789313426) [11371.008067] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [11372.226588] Key type lgssc unregistered [11372.388365] LNet: 471534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11372.391169] LNetError: 471534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11372.404590] LNet: Removed LNI 192.168.202.111@tcp [11372.821101] Key type .llcrypt unregistered [11372.824035] Key type ._llcrypt unregistered [11381.572407] Key type ._llcrypt registered [11381.573438] Key type .llcrypt registered [11381.616257] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [11381.978508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11382.028219] alg: No test for adler32 (adler32-zlib) [11382.923775] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [11383.027917] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [11384.616222] Key type lgssc registered [11385.108424] Lustre: Echo OBD driver; http://www.lustre.org/ [11389.416986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11389.426884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11390.536415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11390.550896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11390.590228] Lustre: lustre-MDT0000: new disk, initializing [11390.647735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11390.661995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11393.041311] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11398.750799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11398.797360] Lustre: 472722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11398.830894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11398.832790] Lustre: Skipped 1 previous similar message [11398.872363] Lustre: lustre-MDT0001: new disk, initializing [11398.923883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11398.943816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11398.948072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11401.178116] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11407.258889] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11411.205203] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11415.454139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11415.643148] Lustre: lustre-OST0000: new disk, initializing [11415.645830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11415.651536] Lustre: 474689:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11415.709239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11419.650922] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11424.280045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11424.287996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11424.354341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11425.650655] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11427.684903] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing lnet_if_list [11430.576090] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11430.595795] LNet: Added LNI 192.168.202.111@tcp10 [8/256/0/180] [11434.078643] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11434.099150] LNet: Added LNI 192.168.202.111@tcp11 [8/256/0/180] [11436.608974] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11436.626396] LNet: Added LNI 192.168.202.111@tcp12 [8/256/0/180] [11438.885909] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11438.908712] LNet: Added LNI 192.168.202.111@tcp13 [8/256/0/180] [11441.460388] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11441.493822] LNet: Added LNI 192.168.202.111@tcp14 [8/256/0/180] [11444.089377] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11444.112540] LNet: Added LNI 192.168.202.111@tcp15 [8/256/0/180] [11446.806076] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11446.823856] LNet: Added LNI 192.168.202.111@tcp16 [8/256/0/180] [11449.808449] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11449.838580] LNet: Added LNI 192.168.202.111@tcp17 [8/256/0/180] [11452.977795] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11452.998354] LNet: Added LNI 192.168.202.111@tcp18 [8/256/0/180] [11455.962995] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11455.987332] LNet: Added LNI 192.168.202.111@tcp19 [8/256/0/180] [11459.254771] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11459.276438] LNet: Added LNI 192.168.202.111@tcp20 [8/256/0/180] [11462.007272] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11462.021764] LNet: Added LNI 192.168.202.111@tcp21 [8/256/0/180] [11464.670650] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11464.689231] LNet: Added LNI 192.168.202.111@tcp22 [8/256/0/180] [11467.553765] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11467.572973] LNet: Added LNI 192.168.202.111@tcp23 [8/256/0/180] [11470.710991] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11470.728789] LNet: Added LNI 192.168.202.111@tcp24 [8/256/0/180] [11473.429526] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11473.447908] LNet: Added LNI 192.168.202.111@tcp25 [8/256/0/180] [11476.570960] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11476.589972] LNet: Added LNI 192.168.202.111@tcp26 [8/256/0/180] [11480.092351] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11480.111715] LNet: Added LNI 192.168.202.111@tcp27 [8/256/0/180] [11483.314336] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11483.333600] LNet: Added LNI 192.168.202.111@tcp28 [8/256/0/180] [11486.327418] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11486.342212] LNet: Added LNI 192.168.202.111@tcp29 [8/256/0/180] [11489.360824] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11489.375725] LNet: Added LNI 192.168.202.111@tcp30 [8/256/0/180] [11492.947987] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11492.961519] LNet: Added LNI 192.168.202.111@tcp31 [8/256/0/180] [11496.522466] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11496.539962] LNet: Added LNI 192.168.202.111@tcp32 [8/256/0/180] [11500.854857] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11500.878943] LNet: Added LNI 192.168.202.111@tcp33 [8/256/0/180] [11505.120689] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11505.139731] LNet: Added LNI 192.168.202.111@tcp34 [8/256/0/180] [11509.183873] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11509.211523] LNet: Added LNI 192.168.202.111@tcp35 [8/256/0/180] [11513.026544] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11513.046182] LNet: Added LNI 192.168.202.111@tcp36 [8/256/0/180] [11517.207221] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11517.233543] LNet: Added LNI 192.168.202.111@tcp37 [8/256/0/180] [11521.355285] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11521.377563] LNet: Added LNI 192.168.202.111@tcp38 [8/256/0/180] [11525.211567] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11525.225784] LNet: Added LNI 192.168.202.111@tcp39 [8/256/0/180] [11529.275618] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11529.296195] LNet: Added LNI 192.168.202.111@tcp40 [8/256/0/180] [11533.202432] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11533.219353] LNet: Added LNI 192.168.202.111@tcp41 [8/256/0/180] [11537.596766] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11537.619613] LNet: Added LNI 192.168.202.111@tcp42 [8/256/0/180] [11542.011156] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11542.031012] LNet: Added LNI 192.168.202.111@tcp43 [8/256/0/180] [11546.543124] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11546.558661] LNet: Added LNI 192.168.202.111@tcp44 [8/256/0/180] [11551.365177] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11551.383294] LNet: Added LNI 192.168.202.111@tcp45 [8/256/0/180] [11556.463471] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11556.479932] LNet: Added LNI 192.168.202.111@tcp46 [8/256/0/180] [11560.403576] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11560.419112] LNet: Added LNI 192.168.202.111@tcp47 [8/256/0/180] [11563.873536] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11563.891172] LNet: Added LNI 192.168.202.111@tcp48 [8/256/0/180] [11567.450692] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11567.464706] LNet: Added LNI 192.168.202.111@tcp49 [8/256/0/180] [11571.399309] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11571.422461] LNet: Added LNI 192.168.202.111@tcp50 [8/256/0/180] [11582.947344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11582.950748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11582.953507] Lustre: Skipped 2 previous similar messages [11582.957159] Lustre: Skipped 1 previous similar message [11588.027201] Lustre: server umount lustre-MDT0000 complete [11589.943855] LustreError: 473516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789313655 with bad export cookie 14070263546937388232 [11589.946690] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11589.951404] LustreError: 473516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11590.159245] Lustre: server umount lustre-MDT0001 complete [11600.294304] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [11604.801270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11609.568165] Lustre: 471993:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789313658/real 1789313658] req@000000004931613b x1876231138164224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789313674 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11609.582151] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11610.592167] Lustre: 471994:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789313659/real 1789313659] req@000000005cc54613 x1876231138164352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789313675 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11614.688134] Lustre: 471995:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789313663/real 1789313663] req@00000000942db862 x1876231138164480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789313679 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11615.712601] LustreError: 471991:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c1bf388c x1876231138165248/t0(0) o250->MGC192.168.202.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11615.833192] LustreError: 474683:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11615.843593] LustreError: 474683:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11615.856204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11616.736376] Lustre: 471993:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789313665/real 1789313665] req@000000005aec0384 x1876231138164736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789313681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11617.676437] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11618.915793] LustreError: 498884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11618.925855] LustreError: 498884:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11620.961105] Lustre: 471993:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789313670/real 1789313670] req@000000009e9087df x1876231138164864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789313686 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11621.848569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11622.055558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11624.173568] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11627.500472] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11628.240261] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11631.513657] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11638.760750] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11641.829925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11641.833193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11641.840810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11642.849970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11642.854545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11647.521205] Lustre: server umount lustre-OST0000 complete [11653.089666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11653.093220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11653.098063] Lustre: Skipped 1 previous similar message [11653.101190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11653.103858] Lustre: Skipped 2 previous similar messages [11655.280429] Lustre: server umount lustre-MDT0000 complete [11656.899603] LustreError: 472715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789313722 with bad export cookie 14070263546937390577 [11656.900942] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11656.903901] LustreError: 472715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11657.029292] Lustre: server umount lustre-MDT0001 complete [11661.632337] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [11663.014858] Key type lgssc unregistered [11663.207181] LNet: 501660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11663.214749] LNetError: 501660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11663.218766] LNet: Removed LNI 192.168.202.111@tcp [11663.221051] LNet: Removed LNI 192.168.202.111@tcp10 [11663.223632] LNet: Removed LNI 192.168.202.111@tcp11 [11663.225991] LNet: Removed LNI 192.168.202.111@tcp12 [11663.228203] LNet: Removed LNI 192.168.202.111@tcp13 [11663.230386] LNet: Removed LNI 192.168.202.111@tcp14 [11663.233390] LNet: Removed LNI 192.168.202.111@tcp15 [11663.235363] LNet: Removed LNI 192.168.202.111@tcp16 [11663.237067] LNet: Removed LNI 192.168.202.111@tcp17 [11663.238816] LNet: Removed LNI 192.168.202.111@tcp18 [11663.240313] LNet: Removed LNI 192.168.202.111@tcp19 [11663.242059] LNet: Removed LNI 192.168.202.111@tcp20 [11663.243991] LNet: Removed LNI 192.168.202.111@tcp21 [11663.245617] LNet: Removed LNI 192.168.202.111@tcp22 [11663.246830] LNet: Removed LNI 192.168.202.111@tcp23 [11663.248087] LNet: Removed LNI 192.168.202.111@tcp24 [11663.250226] LNet: Removed LNI 192.168.202.111@tcp25 [11663.251928] LNet: Removed LNI 192.168.202.111@tcp26 [11663.254402] LNet: Removed LNI 192.168.202.111@tcp27 [11663.256329] LNet: Removed LNI 192.168.202.111@tcp28 [11663.258405] LNet: Removed LNI 192.168.202.111@tcp29 [11663.260474] LNet: Removed LNI 192.168.202.111@tcp30 [11663.262124] LNet: Removed LNI 192.168.202.111@tcp31 [11663.263682] LNet: Removed LNI 192.168.202.111@tcp32 [11663.265405] LNet: Removed LNI 192.168.202.111@tcp33 [11663.267071] LNet: Removed LNI 192.168.202.111@tcp34 [11663.268711] LNet: Removed LNI 192.168.202.111@tcp35 [11663.270074] LNet: Removed LNI 192.168.202.111@tcp36 [11663.272153] LNet: Removed LNI 192.168.202.111@tcp37 [11663.274982] LNet: Removed LNI 192.168.202.111@tcp38 [11663.276606] LNet: Removed LNI 192.168.202.111@tcp39 [11663.277908] LNet: Removed LNI 192.168.202.111@tcp40 [11663.281776] LNet: Removed LNI 192.168.202.111@tcp41 [11663.283330] LNet: Removed LNI 192.168.202.111@tcp42 [11663.284726] LNet: Removed LNI 192.168.202.111@tcp43 [11663.285983] LNet: Removed LNI 192.168.202.111@tcp44 [11663.287492] LNet: Removed LNI 192.168.202.111@tcp45 [11663.288697] LNet: Removed LNI 192.168.202.111@tcp46 [11663.289846] LNet: Removed LNI 192.168.202.111@tcp47 [11663.292290] LNet: Removed LNI 192.168.202.111@tcp48 [11663.294485] LNet: Removed LNI 192.168.202.111@tcp49 [11663.301355] LNet: Removed LNI 192.168.202.111@tcp50 [11663.699917] Key type .llcrypt unregistered [11663.701073] Key type ._llcrypt unregistered [11673.170391] Key type ._llcrypt registered [11673.172048] Key type .llcrypt registered [11673.225281] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [11680.949761] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [11681.344633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11681.389585] alg: No test for adler32 (adler32-zlib) [11682.286597] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [11682.407447] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [11684.008878] Key type lgssc registered [11684.766704] Lustre: Echo OBD driver; http://www.lustre.org/ [11703.243468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11703.258218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11704.377794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11704.396387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11704.445842] Lustre: lustre-MDT0000: new disk, initializing [11704.478953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11704.486195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11706.319735] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11712.498586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11712.573802] Lustre: 504828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11712.604292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11712.606493] Lustre: Skipped 1 previous similar message [11712.645417] Lustre: lustre-MDT0001: new disk, initializing [11712.694033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11712.722620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11712.728974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11714.610972] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11720.069510] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11723.111593] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11726.901887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11726.988803] Lustre: *** cfs_fail_loc=90f, val=0*** [11727.028986] Lustre: lustre-OST0000: new disk, initializing [11727.031630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11727.035979] Lustre: 506842:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11727.077215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11729.744801] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11735.416869] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11736.596084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11736.602544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11736.669942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11741.667254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11741.672434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11741.674942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11741.680712] Lustre: Skipped 1 previous similar message [11741.686435] Lustre: Skipped 1 previous similar message [11745.334522] Lustre: server umount lustre-OST0000 complete [11748.833950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11748.839527] LustreError: Skipped 1 previous similar message [11748.843292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11748.851424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11751.905302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11751.906371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11753.360271] Lustre: server umount lustre-MDT0000 complete [11755.231172] LustreError: 504818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789313820 with bad export cookie 8834782560657725369 [11755.236266] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11755.239801] LustreError: 504818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11755.415171] Lustre: server umount lustre-MDT0001 complete [11760.417222] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [11761.656843] Key type lgssc unregistered [11761.803510] LNet: 508742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11761.806165] LNetError: 508742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11761.815427] LNet: Removed LNI 192.168.202.111@tcp [11762.198166] Key type .llcrypt unregistered [11762.199338] Key type ._llcrypt unregistered [11770.743697] Key type ._llcrypt registered [11770.745303] Key type .llcrypt registered [11770.790256] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [11771.212288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11771.227139] alg: No test for adler32 (adler32-zlib) [11772.099020] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [11772.219294] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [11773.824276] Key type lgssc registered [11774.241823] Lustre: Echo OBD driver; http://www.lustre.org/ [11777.106752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11777.116838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11778.306643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11779.965876] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11783.829402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11784.003722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11785.744490] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11789.118674] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11791.681074] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11794.489532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11794.643274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11796.964733] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11800.027213] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11814.599052] loop0: detected capacity change from 0 to 204800000 [11814.673919] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11814.678724] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11814.686743] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11817.204213] loop0: detected capacity change from 0 to 204800000 [11817.319524] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11817.324689] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11817.330161] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11819.317409] loop0: detected capacity change from 0 to 204800000 [11821.562609] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11821.622923] Lustre: fs15246-OST0000: new disk, initializing [11821.624855] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [11821.627537] Lustre: 513597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11821.650492] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11823.834814] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11828.148904] loop1: detected capacity change from 0 to 204800000 [11830.413584] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11830.455833] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [11830.478170] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [11830.515156] Lustre: fs15246-MDT0000: new disk, initializing [11830.572783] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11830.582697] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11830.624680] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11831.649804] Lustre: 514720:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000141f6055 x1876231545979520/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789313912 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [11832.265680] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11840.738340] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11840.742631] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11840.752498] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11841.417666] Lustre: server umount fs15246-MDT0000 complete [11851.072053] Lustre: server umount fs15246-OST0000 complete [11859.425136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11859.428545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.434609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11860.449475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11860.450078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11860.452416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11861.460726] Lustre: server umount lustre-OST0000 complete [11865.570092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11865.571662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11865.577076] Lustre: Skipped 1 previous similar message [11865.581831] Lustre: Skipped 2 previous similar messages [11870.689442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11870.692913] Lustre: Skipped 1 previous similar message [11875.809143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11875.812025] Lustre: Skipped 1 previous similar message [11877.344114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11877.405791] Lustre: server umount lustre-MDT0000 complete [11878.778180] LustreError: 509965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789313943 with bad export cookie 13646089814902674083 [11878.781266] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11878.782027] LustreError: 509965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11878.912020] Lustre: server umount lustre-MDT0001 complete [11882.946611] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [11884.031628] Key type lgssc unregistered [11884.175456] LNet: 517812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11884.178276] LNetError: 517812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11884.184296] LNet: Removed LNI 192.168.202.111@tcp [11884.512420] Key type .llcrypt unregistered [11884.513632] Key type ._llcrypt unregistered [11891.881344] Key type ._llcrypt registered [11891.882514] Key type .llcrypt registered [11891.928478] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [11892.354760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11892.404107] alg: No test for adler32 (adler32-zlib) [11893.298105] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [11893.410664] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [11895.000245] Key type lgssc registered [11895.456271] Lustre: Echo OBD driver; http://www.lustre.org/ [11911.136186] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [11914.935377] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [11928.469116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11928.475042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11929.560088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11929.575027] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11929.611623] Lustre: lustre-MDT0000: new disk, initializing [11929.637359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11929.645576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11931.153146] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11935.878973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11935.930551] Lustre: 522245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11935.961461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11935.963456] Lustre: Skipped 1 previous similar message [11936.004141] Lustre: lustre-MDT0001: new disk, initializing [11936.045914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11936.067361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11936.071401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11937.548420] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11941.506095] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11943.592427] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11946.115695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11946.229377] Lustre: lustre-OST0000: new disk, initializing [11946.230996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11946.234376] Lustre: 524215:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11946.264465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11948.424185] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11952.399160] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11953.928457] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11954.176353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11954.181844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11954.215497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11955.027713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [11956.460726] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11956.534507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11959.267573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11959.268820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11959.270062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11959.271909] Lustre: Skipped 1 previous similar message [11959.278554] LustreError: Skipped 1 previous similar message [11963.357170] Lustre: server umount lustre-OST0000 complete [11966.945162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11966.948746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11966.953679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11966.955789] Lustre: Skipped 1 previous similar message [11969.505171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11969.505780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11974.625153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11974.628209] Lustre: Skipped 2 previous similar messages [11979.232111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11979.312146] Lustre: server umount lustre-MDT0000 complete [11980.847847] LustreError: 522236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789314046 with bad export cookie 13129907544241326397 [11980.851433] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11980.856719] LustreError: 522236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11980.986658] Lustre: server umount lustre-MDT0001 complete [11989.952617] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [11993.813577] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [12005.837175] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11737 sec ========= 11:41:09 (1789314069) [12006.523189] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:41:10 (1789314070) === [12008.071724] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:41:12 (1789314072) === [12022.852246] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [12023.975706] Key type lgssc unregistered [12024.130599] LNet: 530852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12024.133658] LNetError: 530852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12024.142464] LNet: Removed LNI 192.168.202.111@tcp [12024.575581] Key type .llcrypt unregistered [12024.577535] Key type ._llcrypt unregistered