[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 463016090 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2560MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002361] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006946] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.010121] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.013058] *** VALIDATE selinux *** [ 0.020763] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025481] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028103] *** VALIDATE tmpfs *** [ 0.030289] *** VALIDATE proc *** [ 0.031213] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033254] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034140] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036028] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.040235] debug: unmapping init [mem 0xffffffffad259000-0xffffffffad260fff] [ 0.042166] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043618] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044020] ... version: 2 [ 0.045010] ... bit width: 48 [ 0.046009] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048017] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.049963] ... event mask: 000000070000000f [ 0.050290] rcu: Hierarchical SRCU implementation. [ 0.052433] smp: Bringing up secondary CPUs ... [ 0.053546] x86: Booting SMP configuration: [ 0.054023] .... node #0, CPUs: #1 #2 #3 [ 0.057233] smp: Brought up 1 node, 4 CPUs [ 0.059009] smpboot: Max logical packages: 1 [ 0.060014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142043] node 0 deferred pages initialised in 78ms [ 0.145384] devtmpfs: initialized [ 0.146214] x86/mm: Memory block size: 128MB [ 0.148275] gcov: version magic: 0x41383552 [ 0.151206] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153299] pinctrl core: initialized pinctrl subsystem [ 0.154177] [ 0.154734] ************************************************************* [ 0.155012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156010] ** ** [ 0.157010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158010] ** ** [ 0.159010] ** This means that this kernel is built to expose internal ** [ 0.160009] ** IOMMU data structures, which may compromise security on ** [ 0.161011] ** your system. ** [ 0.162010] ** ** [ 0.163008] ** If you see this message and you are not debugging the ** [ 0.164011] ** kernel, report this immediately to your vendor! ** [ 0.165009] ** ** [ 0.166010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167009] ************************************************************* [ 0.168639] NET: Registered protocol family 16 [ 0.169456] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.172436] cpuidle: using governor menu [ 0.174147] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176419] PCI: Using configuration type 1 for base access [ 0.178146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.187075] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.189155] cryptd: max_cpu_qlen set to 1000 [ 0.190218] ACPI: Added _OSI(Module Device) [ 0.191014] ACPI: Added _OSI(Processor Device) [ 0.192009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.193010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197137] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.200196] ACPI: Interpreter enabled [ 0.201046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.202014] ACPI: Using IOAPIC for interrupt routing [ 0.203086] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.204382] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.213824] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.214032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.215015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.218329] acpiphp: Slot [2] registered [ 0.219089] acpiphp: Slot [5] registered [ 0.220091] acpiphp: Slot [6] registered [ 0.221085] acpiphp: Slot [7] registered [ 0.222087] acpiphp: Slot [8] registered [ 0.223086] acpiphp: Slot [9] registered [ 0.224117] acpiphp: Slot [10] registered [ 0.225097] acpiphp: Slot [3] registered [ 0.226076] acpiphp: Slot [4] registered [ 0.227082] acpiphp: Slot [11] registered [ 0.228078] acpiphp: Slot [12] registered [ 0.229071] acpiphp: Slot [13] registered [ 0.230126] acpiphp: Slot [14] registered [ 0.231084] acpiphp: Slot [15] registered [ 0.232081] acpiphp: Slot [16] registered [ 0.233070] acpiphp: Slot [17] registered [ 0.234074] acpiphp: Slot [18] registered [ 0.235090] acpiphp: Slot [19] registered [ 0.236076] acpiphp: Slot [20] registered [ 0.237105] acpiphp: Slot [21] registered [ 0.238071] acpiphp: Slot [22] registered [ 0.239113] acpiphp: Slot [23] registered [ 0.242137] acpiphp: Slot [24] registered [ 0.244112] acpiphp: Slot [25] registered [ 0.245109] acpiphp: Slot [26] registered [ 0.247201] acpiphp: Slot [27] registered [ 0.248093] acpiphp: Slot [28] registered [ 0.250096] acpiphp: Slot [29] registered [ 0.252099] acpiphp: Slot [30] registered [ 0.253083] acpiphp: Slot [31] registered [ 0.255118] PCI host bridge to bus 0000:00 [ 0.256021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.259081] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.261021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.264025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.266030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.269026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.271187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.274022] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.277378] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.287781] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.294013] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.296018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.299016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.301021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.303719] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.307943] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.309027] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.311806] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.316013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.330015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.335014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.339292] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.352015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.360014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.377016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.384843] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.390012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.395046] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.410017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.417577] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.423014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.427016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.441018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.452171] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.461012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.469018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.488016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.497964] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.508016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.515016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.529018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.539359] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.544014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.549014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.570017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.577437] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.579286] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.581310] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.583244] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.585179] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.588125] iommu: Default domain type: Passthrough [ 0.590228] SCSI subsystem initialized [ 0.591000] ACPI: bus type USB registered [ 0.593132] usbcore: registered new interface driver usbfs [ 0.594045] usbcore: registered new interface driver hub [ 0.596053] usbcore: registered new device driver usb [ 0.597095] pps_core: LinuxPPS API ver. 1 registered [ 0.598008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.601071] PTP clock support registered [ 0.603128] EDAC MC: Ver: 3.0.0 [ 0.604389] PCI: Using ACPI for IRQ routing [ 0.606073] NetLabel: Initializing [ 0.607013] NetLabel: domain hash size = 128 [ 0.608007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.609169] NetLabel: unlabeled traffic allowed by default [ 0.611171] vgaarb: loaded [ 0.613247] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.615010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.622000] clocksource: Switched to clocksource kvm-clock [ 0.714511] VFS: Disk quotas dquot_6.6.0 [ 0.715618] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.717822] *** VALIDATE ramfs *** [ 0.718803] *** VALIDATE hugetlbfs *** [ 0.719888] pnp: PnP ACPI init [ 0.721646] pnp: PnP ACPI: found 6 devices [ 0.736796] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.739390] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.740989] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.742781] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.744771] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.746934] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.748831] NET: Registered protocol family 2 [ 0.750638] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.754844] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.757669] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.761678] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.764154] TCP: Hash tables configured (established 65536 bind 65536) [ 0.766088] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.768164] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.770144] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.772355] NET: Registered protocol family 1 [ 0.774083] RPC: Registered named UNIX socket transport module. [ 0.775516] RPC: Registered udp transport module. [ 0.776864] RPC: Registered tcp transport module. [ 0.778593] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.780399] NET: Registered protocol family 44 [ 0.781651] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.783204] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.784828] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.786606] PCI: CLS 0 bytes, default 64 [ 0.789117] Unpacking initramfs... [ 2.168150] debug: unmapping init [mem 0xffff9fe37cc54000-0xffff9fe37ffbffff] [ 2.172599] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.175218] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.178332] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.629130] Initialise system trusted keyrings [ 2.630315] Key type blacklist registered [ 2.631714] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.638935] zbud: loaded [ 2.641249] *** VALIDATE nfs *** [ 2.642115] *** VALIDATE nfs4 *** [ 2.643194] pstore: using deflate compression [ 2.645889] Platform Keyring initialized [ 2.726860] NET: Registered protocol family 38 [ 2.728383] Key type asymmetric registered [ 2.729343] Asymmetric key parser 'x509' registered [ 2.730411] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.732327] io scheduler mq-deadline registered [ 2.733550] io scheduler kyber registered [ 2.734752] io scheduler bfq registered [ 2.735952] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.738352] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.740230] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.742132] ACPI: Power Button [PWRF] [ 2.745962] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.750589] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.770886] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.779415] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.802991] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.828498] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.855475] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.858810] Non-volatile memory driver v1.3 [ 2.859892] Linux agpgart interface v0.103 [ 2.886118] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 2.888128] vda: detected capacity change from 0 to 74829824 [ 2.911672] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.914907] vdb: detected capacity change from 0 to 1073741824 [ 2.936706] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.938592] vdc: detected capacity change from 0 to 2621440000 [ 2.952714] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.954696] vdd: detected capacity change from 0 to 2621440000 [ 2.967953] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.970348] vde: detected capacity change from 0 to 4294967296 [ 2.983374] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.986064] vdf: detected capacity change from 0 to 4294967296 [ 2.993708] libphy: Fixed MDIO Bus: probed [ 3.001604] usbcore: registered new interface driver usbserial_generic [ 3.004014] usbserial: USB Serial support registered for generic [ 3.006402] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.010896] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.012691] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.015134] mousedev: PS/2 mouse device common for all mice [ 3.018239] rtc_cmos 00:05: RTC can wake from S4 [ 3.021116] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.025914] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.029469] rtc_cmos 00:05: registered as rtc0 [ 3.030834] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.031592] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.031628] intel_pstate: CPU model not supported [ 3.037385] hid: raw HID events driver (C) Jiri Kosina [ 3.037568] usbcore: registered new interface driver usbhid [ 3.042890] usbhid: USB HID core driver [ 3.044564] drop_monitor: Initializing network drop monitor service [ 3.046602] Initializing XFRM netlink socket [ 3.048615] NET: Registered protocol family 10 [ 3.051762] Segment Routing with IPv6 [ 3.053311] NET: Registered protocol family 17 [ 3.055330] mpls_gso: MPLS GSO support [ 3.060487] RAS: Correctable Errors collector initialized. [ 3.062702] AVX version of gcm_enc/dec engaged. [ 3.064190] AES CTR mode by8 optimization enabled [ 3.119154] sched_clock: Marking stable (3119098514, 0)->(3991756866, -872658352) [ 3.121703] registered taskstats version 1 [ 3.122922] Loading compiled-in X.509 certificates [ 3.124122] zswap: loaded using pool lzo/zbud [ 3.144316] Key type big_key registered [ 3.155768] Key type encrypted registered [ 3.156988] ima: No TPM chip found, activating TPM-bypass! [ 3.158831] ima: Allocated hash algorithm: sha1 [ 3.160140] ima: No architecture policies found [ 3.161308] evm: Initialising EVM extended attributes: [ 3.162517] evm: security.selinux [ 3.163636] evm: security.ima [ 3.164503] evm: security.capability [ 3.165289] evm: HMAC attrs: 0x1 [ 3.166910] rtc_cmos 00:05: setting system clock to 2026-08-24 13:52:49 UTC (1787579569) [ 3.171231] debug: unmapping init [mem 0xffffffffae203000-0xffffffffae3fffff] [ 3.173280] debug: unmapping init [mem 0xffffffffacf82000-0xffffffffad258fff] [ 3.181050] Write protecting the kernel read-only data: 28672k [ 3.182917] debug: unmapping init [mem 0xffffffffab603000-0xffffffffab7fffff] [ 3.185062] debug: unmapping init [mem 0xffffffffabf14000-0xffffffffabffffff] [ 3.209200] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.214261] systemd[1]: Detected virtualization kvm. [ 3.215589] systemd[1]: Detected architecture x86-64. [ 3.216699] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.243167] systemd[1]: No hostname configured. [ 3.244308] systemd[1]: Set hostname to . [ 3.245574] random: systemd: uninitialized urandom read (16 bytes read) [ 3.247152] systemd[1]: Initializing machine ID from random generator. [ 3.297460] random: ln: uninitialized urandom read (6 bytes read) [ 3.382949] random: systemd: uninitialized urandom read (16 bytes read) [ 3.385274] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.390228] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.396373] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.931736] device-mapper: uevent: version 1.0.3 [ 3.933892] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.589982] virtio_net virtio0 ens2: renamed from eth0 [ 4.621948] scsi host0: ata_piix [ 4.635189] scsi host1: ata_piix [ 4.635980] random: fast init done [ 4.638663] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.642760] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.534447] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.669064] random: crng init done [ 9.671956] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.076374] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.248688] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.494183] SELinux: Disabled at runtime. [ 11.561364] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.570392] systemd[1]: Detected virtualization kvm. [ 11.572484] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.039255] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.043018] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.048303] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.056551] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.060940] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.069958] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.074566] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [[ 12.119851] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.545376] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.859423] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.912475] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.082534] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.091357] EDAC sbridge: Ver: 1.1.2 [ 14.782906] Key type dns_resolver registered [ 15.082696] NFS: Registering the id_resolver key type [ 15.084450] Key type id_resolver registered [ 15.085790] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg136-server login: [ 56.231155] hrtimer: interrupt took 8446244 ns [ 69.057763] libcfs: loading out-of-tree module taints kernel. [ 69.094867] Key type ._llcrypt registered [ 69.096474] Key type .llcrypt registered [ 69.169750] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 90.363715] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 92.072898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 92.084826] alg: No test for adler32 (adler32-zlib) [ 93.663869] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 94.770787] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 96.599181] Key type lgssc registered [ 98.627581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 122.237700] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 170.770551] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 192.328054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 192.382182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.873284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.946990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 194.074426] Lustre: lustre-MDT0000: new disk, initializing [ 194.198495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.268641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 199.391987] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.735152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.879384] Lustre: 6524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 215.937537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 215.941157] Lustre: Skipped 1 previous similar message [ 216.039891] Lustre: lustre-MDT0001: new disk, initializing [ 216.106195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.135768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 216.157535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 221.123883] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.552099] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 237.920509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.161832] Lustre: lustre-OST0000: new disk, initializing [ 238.165297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 238.174618] Lustre: 8459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.263342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 245.267631] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.843699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 247.860756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 247.960758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 261.478516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.614304] Lustre: lustre-OST0001: new disk, initializing [ 261.618518] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 261.623755] Lustre: 9533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 261.681781] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 267.611672] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.411272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 271.425208] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 271.536425] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 279.769742] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 286.959728] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 293.654804] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing check_logdir /tmp/testlogs/ [ 298.888315] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing yml_node [ 303.617956] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 306.268511] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 309.281784] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 311.938874] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 09:57:56 EDT 2026 [ 334.828854] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 336.972725] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 353.768887] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.784575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 353.786478] Lustre: Skipped 1 previous similar message [ 353.810121] Lustre: Skipped 3 previous similar messages [ 358.971875] Lustre: server umount lustre-MDT0000 complete [ 363.009940] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579929 with bad export cookie 12291373064925543795 [ 363.010700] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 363.019351] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 364.000666] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.003347] LustreError: 6529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.005696] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 364.025208] Lustre: Skipped 3 previous similar messages [ 364.071390] LustreError: 6529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 369.125794] LustreError: 6530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.126316] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 369.163670] Lustre: Skipped 2 previous similar messages [ 369.460304] Lustre: server umount lustre-MDT0001 complete [ 373.474140] Lustre: server umount lustre-OST0000 complete [ 377.556205] Lustre: server umount lustre-OST0001 complete [ 384.207279] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 394.109618] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 431.676040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.006306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 432.057490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 432.188972] Lustre: lustre-MDT0000: new disk, initializing [ 432.308824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.334493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 437.145541] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 449.918704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.114391] Lustre: 15088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 450.226474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 450.230738] Lustre: Skipped 1 previous similar message [ 450.358799] Lustre: lustre-MDT0001: new disk, initializing [ 450.437612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 450.479627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 450.495795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 455.238974] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 465.562063] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 471.106351] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 478.305226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 478.660018] Lustre: lustre-OST0000: new disk, initializing [ 478.667864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 478.673889] Lustre: 17055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 478.796476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 480.695627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 480.711473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 480.969298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 485.287125] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 498.177484] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 504.511250] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 504.788409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 511.093303] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 511.347506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 516.584733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 516.598849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.645810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 516.662606] Lustre: Skipped 1 previous similar message [ 519.677217] Lustre: server umount lustre-OST0000 complete [ 526.818889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.824615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 526.839408] Lustre: Skipped 2 previous similar messages [ 530.212319] Lustre: server umount lustre-MDT0000 complete [ 531.944047] LustreError: 15100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.968118] LustreError: 15100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 533.986144] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580100 with bad export cookie 12291373064925545328 [ 533.994156] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 534.005796] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 534.549498] Lustre: server umount lustre-MDT0001 complete [ 539.949522] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:01:44 (1787580104) [ 542.484868] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 543.901313] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:01:49 (1787580109) [ 550.522298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.010900] LustreError: 19480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.083531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.368973] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 556.530502] LustreError: 19481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.582311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.038920] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 578.804181] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 585.494687] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 592.648750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 599.418945] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 608.142035] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 622.049308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 622.056218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.073765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 622.083514] Lustre: Skipped 2 previous similar messages [ 625.676500] Lustre: server umount lustre-OST0000 complete [ 632.289095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.304214] Lustre: Skipped 1 previous similar message [ 636.427292] Lustre: server umount lustre-MDT0000 complete [ 637.418928] LustreError: 21763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.446851] LustreError: 21763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 640.635848] LustreError: 19460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580206 with bad export cookie 12291373064925546210 [ 640.649015] LustreError: 19460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 640.651980] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.966816] Lustre: server umount lustre-MDT0001 complete [ 653.829292] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 656.477641] Key type lgssc unregistered [ 656.694801] LNet: 23018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.705212] LNetError: 23018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.725963] LNet: Removed LNI 192.168.201.136@tcp [ 657.807194] Key type .llcrypt unregistered [ 657.810213] Key type ._llcrypt unregistered [ 682.532438] Key type ._llcrypt registered [ 682.537416] Key type .llcrypt registered [ 682.663428] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 683.665417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 683.928943] alg: No test for adler32 (adler32-zlib) [ 685.019942] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 685.200939] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 686.863202] Key type lgssc registered [ 688.358664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 698.995318] loop: module loaded [ 699.119538] loop0: detected capacity change from 0 to 204800000 [ 699.426093] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 699.457409] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 700.001906] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 708.268531] loop0: detected capacity change from 0 to 204800000 [ 708.506931] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 708.514119] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 708.534902] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.721269] loop0: detected capacity change from 0 to 204800000 [ 722.808497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 722.869641] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 724.186176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 724.309021] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 724.393472] Lustre: lustre-MDT0000: new disk, initializing [ 724.535706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 724.567676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 729.631765] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 738.908739] loop1: detected capacity change from 0 to 204800000 [ 744.726285] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 745.119833] Lustre: lustre-OST07c6: new disk, initializing [ 745.123794] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 745.127507] Lustre: Skipped 1 previous similar message [ 745.131136] Lustre: 26212:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 745.200361] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 750.461972] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 753.763914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 753.790784] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 753.840954] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 755.500557] Lustre: 25198:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 755.509426] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 755.515586] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 755.533050] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 755.543497] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 755.560300] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 772.335796] Lustre: 25198:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 772.343414] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 772.348746] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.357269] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 772.361742] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 772.366721] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.203445] Lustre: 26219:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 777.216631] Lustre: 26219:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 777.228662] Lustre: 26219:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 777.238286] Lustre: 26219:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.247874] Lustre: 26219:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 777.261478] Lustre: 26219:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.273817] Lustre: 26219:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 777.281648] Lustre: 26219:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.286832] Lustre: 26219:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 777.292758] Lustre: 26219:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.303253] Lustre: 26219:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.311328] Lustre: 26219:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.585657] Lustre: 25197:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 780.601398] Lustre: 25197:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 780.611611] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 780.627648] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.635703] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 780.640984] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.646140] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 780.651169] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.655592] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 780.659750] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.665217] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 780.670501] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.563655] Lustre: 25198:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 785.572252] Lustre: 25198:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 785.584675] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 785.591275] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.599870] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 785.608479] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.613960] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 785.622049] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.630098] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 785.635832] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.640817] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 785.645083] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.439331] Lustre: 26219:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 797.456900] Lustre: 26219:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 797.468633] Lustre: 26219:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 797.478222] Lustre: 26219:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 797.489381] Lustre: 26219:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 797.500573] Lustre: 26219:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 797.507314] Lustre: 26219:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 797.526100] Lustre: 26219:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 797.546370] Lustre: 26219:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 797.555396] Lustre: 26219:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 797.564331] Lustre: 26219:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 797.573799] Lustre: 26219:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 845.793042] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 845.818417] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 850.977126] Lustre: server umount lustre-OST07c6 complete [ 860.973370] Lustre: server umount lustre-MDT0000 complete [ 880.211492] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 882.715651] Key type lgssc unregistered [ 883.050073] LNet: 28771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.067762] LNetError: 28771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.098463] LNet: Removed LNI 192.168.201.136@tcp [ 883.953039] Key type .llcrypt unregistered [ 883.954856] Key type ._llcrypt unregistered [ 904.756938] Key type ._llcrypt registered [ 904.763929] Key type .llcrypt registered [ 904.979323] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 906.203116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 906.250402] alg: No test for adler32 (adler32-zlib) [ 907.432512] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 907.654416] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 909.330664] Key type lgssc registered [ 910.595229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 919.420355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 919.465806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 920.955065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 927.207987] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 931.300354] LustreError: 30046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 931.332025] LustreError: 30046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 936.431719] LustreError: 30045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 938.688817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 939.261577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 945.686983] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 956.076460] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 963.193132] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 970.899982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 971.265850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 979.677422] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 989.232530] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 990.803477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1010.147043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1010.160620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.175339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1011.169491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1011.182635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1011.190226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1011.217375] Lustre: Skipped 1 previous similar message [ 1013.113903] Lustre: server umount lustre-OST0000 complete [ 1021.413468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1021.421910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1021.425552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1021.449743] Lustre: Skipped 1 previous similar message [ 1023.449495] Lustre: server umount lustre-MDT0000 complete [ 1026.533036] LustreError: 31586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.557248] LustreError: 31586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1027.388438] LustreError: 30026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580593 with bad export cookie 6547612991921325062 [ 1027.399378] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1027.402467] LustreError: 30026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1027.718452] Lustre: server umount lustre-MDT0001 complete [ 1040.148126] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1043.091566] Key type lgssc unregistered [ 1043.428741] LNet: 33788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.443648] LNetError: 33788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.463133] LNet: Removed LNI 192.168.201.136@tcp [ 1044.498655] Key type .llcrypt unregistered [ 1044.501078] Key type ._llcrypt unregistered [ 1063.535277] Key type ._llcrypt registered [ 1063.540134] Key type .llcrypt registered [ 1063.684886] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1064.825915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1064.937487] alg: No test for adler32 (adler32-zlib) [ 1066.062617] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 1066.363472] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1068.023231] Key type lgssc registered [ 1069.280770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1076.751747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1076.774915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.189362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.616397] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1088.482065] LustreError: 35075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1088.511835] LustreError: 35075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1089.724543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.062730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1093.931473] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1101.365737] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1107.375699] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1114.086974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1114.354430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.210452] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1123.582238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1131.388751] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1146.344751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1146.357653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.414653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1149.408560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1149.419606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.423433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1150.795466] Lustre: server umount lustre-OST0000 complete [ 1156.576745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1156.584850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.601702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1156.616389] Lustre: Skipped 1 previous similar message [ 1159.648202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1159.648652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.080863] Lustre: server umount lustre-MDT0000 complete [ 1164.772502] LustreError: 35075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.815982] LustreError: 35075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1166.202718] LustreError: 35055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580732 with bad export cookie 3626818360332375421 [ 1166.237427] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1166.238199] LustreError: 35055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1166.855441] Lustre: server umount lustre-MDT0001 complete [ 1177.719827] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1180.321838] Key type lgssc unregistered [ 1180.598710] LNet: 38624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1180.613423] LNetError: 38624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1180.631378] LNet: Removed LNI 192.168.201.136@tcp [ 1181.531737] Key type .llcrypt unregistered [ 1181.534097] Key type ._llcrypt unregistered [ 1205.479667] Key type ._llcrypt registered [ 1205.482864] Key type .llcrypt registered [ 1205.639530] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1206.693986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1206.867944] alg: No test for adler32 (adler32-zlib) [ 1207.930719] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 1208.134944] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1209.823368] Key type lgssc registered [ 1211.080513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1219.287684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1219.304948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.685618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.282397] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1229.802980] LustreError: 40044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1229.825853] LustreError: 40044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1234.410326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.795116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1239.679507] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1247.429841] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1253.754970] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1261.177551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1261.483435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1265.600490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1268.153360] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1276.629864] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1286.122568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1286.140510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.187461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1288.934042] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 1290.221062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.225066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.255707] Lustre: Skipped 1 previous similar message [ 1290.286648] Lustre: Skipped 2 previous similar messages [ 1292.535581] Lustre: server umount lustre-MDT0000 complete [ 1294.033512] LustreError: 40039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.346024] LustreError: 41049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.365581] LustreError: 41049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1297.101655] LustreError: 40024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580863 with bad export cookie 3983554414634804718 [ 1297.114905] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1299.145310] LustreError: 40039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.181219] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [ 1300.450043] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.573784] Lustre: server umount lustre-MDT0001 complete [ 1319.377048] Lustre: server umount lustre-OST0000 complete [ 1337.545929] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1340.594872] Key type lgssc unregistered [ 1340.861918] LNet: 43917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1340.869375] LNetError: 43917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1341.933907] LNet: Removed LNI 192.168.201.136@tcp [ 1343.026969] Key type .llcrypt unregistered [ 1343.028755] Key type ._llcrypt unregistered [ 1367.365802] Key type ._llcrypt registered [ 1367.367027] Key type .llcrypt registered [ 1367.503376] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1368.733393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1368.811705] alg: No test for adler32 (adler32-zlib) [ 1369.975149] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 1370.140071] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1371.775225] Key type lgssc registered [ 1373.066967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1381.073553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1381.093424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.503622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.519736] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1392.109867] LustreError: 45332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1392.146720] LustreError: 45332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1397.221100] LustreError: 45333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1397.772800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1398.261702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1403.346204] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1411.945572] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1418.342433] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1425.987338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1426.330237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.401630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1432.835586] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1441.289910] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1449.200799] Lustre: lustre-OST0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 1453.034813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1453.035739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.068074] LustreError: Skipped 1 previous similar message [ 1453.073172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1453.105991] Lustre: Skipped 1 previous similar message [ 1454.278963] Lustre: lustre-OST0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 1454.283735] Lustre: Skipped 1 previous similar message [ 1458.145786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1458.155671] Lustre: Skipped 1 previous similar message [ 1462.241731] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1463.285092] LustreError: 47154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.145250] Lustre: server umount lustre-OST0000 complete [ 1484.143156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1484.147940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1484.177228] Lustre: Skipped 1 previous similar message [ 1484.201661] Lustre: Skipped 1 previous similar message [ 1490.815900] Lustre: server umount lustre-MDT0000 complete [ 1493.476815] LustreError: 46061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1493.503183] LustreError: 46061:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1495.177630] LustreError: 45314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581061 with bad export cookie 7696151551560103110 [ 1495.186760] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1495.189625] LustreError: 45314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1501.801806] Lustre: server umount lustre-MDT0001 complete [ 1513.870526] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1516.617595] Key type lgssc unregistered [ 1516.969401] LNet: 49045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1516.982310] LNetError: 49045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.021326] LNet: Removed LNI 192.168.201.136@tcp [ 1517.974373] Key type .llcrypt unregistered [ 1517.977101] Key type ._llcrypt unregistered [ 1542.785745] Key type ._llcrypt registered [ 1542.791186] Key type .llcrypt registered [ 1542.914677] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1544.214992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1544.400504] alg: No test for adler32 (adler32-zlib) [ 1545.529909] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 1545.851785] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1547.495185] Key type lgssc registered [ 1548.711374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1557.745516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1557.788347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1559.572488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1564.811794] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1568.756630] LustreError: 50465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.790106] LustreError: 50465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1573.865631] LustreError: 50466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1575.982600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1576.549625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1582.159948] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1591.011924] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1598.123955] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1605.368904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1605.792328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1612.894080] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1621.349907] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1629.364851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1637.070237] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1641.050532] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1653.581361] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2410h20m17s [ 1655.389993] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1658.107466] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 1658.337832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1658.362830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.142428] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 1659.147906] Lustre: Skipped 3 previous similar messages [ 1663.180537] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 1663.198889] Lustre: Skipped 3 previous similar messages [ 1663.591939] Lustre: server umount lustre-MDT0000 complete [ 1665.511880] LustreError: 51610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1665.529561] LustreError: 51610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1666.886984] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1669.319501] LustreError: 51610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1674.487583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1674.689057] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1674.995058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.075737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1676.033126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1679.998750] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1683.993373] Lustre: DEBUG MARKER: Wait for df (36785) ... [ 1686.015122] Lustre: DEBUG MARKER: done [ 1690.595453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1690.600994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.616431] Lustre: Skipped 2 previous similar messages [ 1690.624408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1691.624926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.800237] Lustre: server umount lustre-OST0000 complete [ 1701.863206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.865758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1701.875108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1701.881952] Lustre: Skipped 1 previous similar message [ 1701.900475] Lustre: Skipped 3 previous similar messages [ 1706.452929] Lustre: server umount lustre-MDT0000 complete [ 1709.034662] LustreError: 50465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1709.046764] LustreError: 50465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1710.637246] LustreError: 50444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581276 with bad export cookie 1776921242111782751 [ 1710.638209] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1710.651290] LustreError: 50444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1711.110796] Lustre: server umount lustre-MDT0001 complete [ 1723.875179] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1726.848895] Key type lgssc unregistered [ 1727.130866] LNet: 55418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.138892] LNetError: 55418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.158870] LNet: Removed LNI 192.168.201.136@tcp [ 1727.998191] Key type .llcrypt unregistered [ 1728.000467] Key type ._llcrypt unregistered [ 1775.632914] Key type ._llcrypt registered [ 1775.636146] Key type .llcrypt registered [ 1775.804337] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1777.506304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1777.575334] alg: No test for adler32 (adler32-zlib) [ 1778.799322] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 1779.167626] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1780.943387] Key type lgssc registered [ 1782.561258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1790.898710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1790.930748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1792.275631] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1792.337469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1792.529250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.648485] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1808.256287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1808.499265] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1808.602950] Lustre: 57628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1809.098394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1815.975313] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1828.676748] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1836.543938] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1845.292672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1845.545287] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1845.923361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.299839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1855.197289] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1871.873565] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1881.449143] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2410h24m05s [ 1884.411976] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1893.772731] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1896.302272] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1896.529646] LustreError: 57635:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1916.668123] Lustre: lustre-MDT0000: Client 25e8b592-5a0a-4f4f-ae9f-2c423a682be5 (at 192.168.201.36@tcp) reconnecting [ 1936.551309] LustreError: 57635:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1938.714552] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1941.320881] Lustre: DEBUG MARKER: done [ 1951.726701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1951.731903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.739192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1951.742478] Lustre: Skipped 1 previous similar message [ 1954.371937] Lustre: server umount lustre-OST0000 complete [ 1961.952679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.971864] Lustre: Skipped 1 previous similar message [ 1961.982249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1963.045741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1963.064017] Lustre: Skipped 2 previous similar messages [ 1964.871769] Lustre: server umount lustre-MDT0000 complete [ 1969.248400] LustreError: 57620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581535 with bad export cookie 2442977486007344223 [ 1969.254587] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1969.276914] LustreError: 57620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1969.780489] Lustre: server umount lustre-MDT0001 complete [ 1983.336828] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1986.222628] Key type lgssc unregistered [ 1986.454420] LNet: 61971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1986.459639] LNetError: 61971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1986.479736] LNet: Removed LNI 192.168.201.136@tcp [ 1987.356823] Key type .llcrypt unregistered [ 1987.363792] Key type ._llcrypt unregistered [ 2036.094407] Key type ._llcrypt registered [ 2036.096540] Key type .llcrypt registered [ 2036.197987] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2037.396237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2037.660080] alg: No test for adler32 (adler32-zlib) [ 2038.843135] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 2039.162069] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2040.903200] Key type lgssc registered [ 2041.851929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2045.479744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2045.496123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.768522] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2046.781198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2047.058377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.208356] Lustre: Failing over lustre-MDT0000 [ 2048.543043] Lustre: server umount lustre-MDT0000 complete [ 2057.029805] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:27:01 (1787581621) [ 2063.822399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.499432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.124870] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2079.686467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.808635] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2079.873694] Lustre: 64771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2080.240799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2085.758332] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2094.532769] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2102.949946] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2112.853140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2113.375474] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2113.692594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2120.955700] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2123.477142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2132.079831] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2141.143828] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2149.353267] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.374167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.383902] Lustre: Skipped 2 previous similar messages [ 2149.414793] Lustre: Skipped 2 previous similar messages [ 2154.467244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2154.476158] Lustre: Skipped 2 previous similar messages [ 2159.593523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.595414] Lustre: Skipped 2 previous similar messages [ 2163.167312] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2163.409414] Lustre: server umount lustre-MDT0000 complete [ 2167.722283] LustreError: 64763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581734 with bad export cookie 5729222145243813401 [ 2167.728557] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2167.735726] LustreError: 64763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2168.043277] Lustre: server umount lustre-MDT0001 complete [ 2171.932549] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2176.547564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2185.779695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.207174] Lustre: 63694:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787581736/real 1787581736] req@0000000036713cbc x1874415169089792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787581752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2186.239531] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.281187] Lustre: 63697:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787581739/real 1787581739] req@0000000019fd314f x1874415169089920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787581755 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2191.391394] Lustre: 63697:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787581741/real 1787581741] req@00000000905ba2b5 x1874415169090048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787581757 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2194.117530] LustreError: 66623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.143964] LustreError: 66623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2194.251723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.399478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2195.176740] Lustre: 63694:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787581744/real 1787581744] req@0000000026b7863d x1874415169090304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787581760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2195.308860] LustreError: 66624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.320032] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2196.389050] LustreError: 68544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2196.414403] LustreError: 68544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2199.345365] Lustre: 63694:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787581749/real 1787581749] req@0000000017d2b041 x1874415169090432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787581765 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2200.333710] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2201.573062] LustreError: 68543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2201.600692] LustreError: 68543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2206.693239] LustreError: 68542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.715254] LustreError: 68542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2210.151000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.495623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2215.922923] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2216.169033] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2224.995189] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2233.297579] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2241.508774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2241.511734] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.538756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2241.553867] Lustre: Skipped 1 previous similar message [ 2241.569539] Lustre: Skipped 2 previous similar messages [ 2246.628628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2246.643093] Lustre: Skipped 2 previous similar messages [ 2247.548423] Lustre: server umount lustre-MDT0000 complete [ 2251.750129] LustreError: 68543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.764953] LustreError: 68543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2252.335745] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581818 with bad export cookie 5729222145243815676 [ 2252.340836] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2252.345713] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2252.707254] Lustre: server umount lustre-MDT0001 complete [ 2260.688065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2271.146114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.247209] Lustre: 63695:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787581823/real 1787581823] req@00000000a232b1f7 x1874415169120128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787581839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2273.283832] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.299371] Lustre: Skipped 1 previous similar message [ 2277.407745] LustreError: 63693:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a232b1f7 x1874415169121024/t0(0) o250->MGC192.168.201.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2277.784160] LustreError: 66623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.879786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2277.963234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2281.957644] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2282.644401] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2291.029471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2296.822413] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2297.480434] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2306.046651] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2313.171196] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2322.402168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2322.403321] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2322.417232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2322.455605] Lustre: Skipped 2 previous similar messages [ 2325.680955] Lustre: server umount lustre-MDT0000 complete [ 2327.526551] LustreError: 71226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.540742] LustreError: 71226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2330.826665] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581897 with bad export cookie 5729222145243817055 [ 2330.838763] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2330.841411] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2331.261719] Lustre: server umount lustre-MDT0001 complete [ 2338.540649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2342.414594] Lustre: DEBUG MARKER: files compared the same [ 2354.170157] Lustre: server umount lustre-OST0000 complete [ 2372.372371] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2375.672143] Key type lgssc unregistered [ 2375.982131] LNet: 74509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2375.990586] LNetError: 74509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2377.071441] LNet: Removed LNI 192.168.201.136@tcp [ 2378.139529] Key type .llcrypt unregistered [ 2378.146078] Key type ._llcrypt unregistered [ 2407.184472] Key type ._llcrypt registered [ 2407.186126] Key type .llcrypt registered [ 2407.378588] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2408.942609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2409.065565] alg: No test for adler32 (adler32-zlib) [ 2410.966310] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 2411.520085] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2413.439774] Key type lgssc registered [ 2415.320724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2425.368325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2425.416146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2427.027888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2432.807353] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug +malloc all [ 2437.603301] LustreError: 75927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.624537] LustreError: 75927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2442.729259] LustreError: 75928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.973267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2445.231673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2450.576347] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug +malloc all [ 2461.293230] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2469.161724] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2478.208793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2478.636688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2486.940567] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug +malloc all [ 2494.411488] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2497.061179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2510.303991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2510.317994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.336470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2512.355119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2512.364444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2512.365305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2512.416574] Lustre: Skipped 1 previous similar message [ 2513.982647] Lustre: server umount lustre-OST0000 complete [ 2522.080234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2522.092308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2522.107183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2524.488338] Lustre: server umount lustre-MDT0000 complete [ 2527.712635] LustreError: 75928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.726025] LustreError: 75928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2529.209096] LustreError: 75907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582095 with bad export cookie 16589593705982392247 [ 2529.220238] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2529.229288] LustreError: 75907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2529.708324] Lustre: server umount lustre-MDT0001 complete [ 2543.962521] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2547.533671] Key type lgssc unregistered [ 2547.790687] LNet: 79492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2547.795531] LNetError: 79492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.849205] LNet: Removed LNI 192.168.201.136@tcp [ 2550.052716] Key type .llcrypt unregistered [ 2550.057106] Key type ._llcrypt unregistered [ 2575.822571] Key type ._llcrypt registered [ 2575.827964] Key type .llcrypt registered [ 2575.928568] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2576.987523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2577.076109] alg: No test for adler32 (adler32-zlib) [ 2578.094936] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 2578.339498] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2580.055456] Key type lgssc registered [ 2581.336820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2590.209718] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2590.218688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2605.731064] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2610.847335] LustreError: 80886:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.136@tcp: failed processing log, type 4: rc = -110 [ 2637.471314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.365118] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2652.094926] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2659.985398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2660.049980] Lustre: *** cfs_fail_loc=706, val=0*** [ 2660.492602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2660.558505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2665.353987] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2665.964283] LustreError: 80912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.988425] LustreError: 80912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2671.083034] LustreError: 80911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.120193] LustreError: 80912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.145025] LustreError: 80912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2674.234643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.516197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2678.973445] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2688.224093] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2695.432265] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2700.273610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2700.278519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.309705] Lustre: Skipped 1 previous similar message [ 2700.312194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2704.594782] Lustre: server umount lustre-OST0000 complete [ 2710.499827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2710.506612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2710.543329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2710.551662] Lustre: Skipped 2 previous similar messages [ 2714.872916] Lustre: server umount lustre-MDT0000 complete [ 2715.617316] LustreError: 81975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.629242] LustreError: 81975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2718.849251] LustreError: 80893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582285 with bad export cookie 18137398825407540798 [ 2718.883935] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2719.114722] Lustre: server umount lustre-MDT0001 complete [ 2729.992547] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2733.128266] Key type lgssc unregistered [ 2733.379683] LNet: 84502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2733.391857] LNetError: 84502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2733.412392] LNet: Removed LNI 192.168.201.136@tcp [ 2734.275771] Key type .llcrypt unregistered [ 2734.278460] Key type ._llcrypt unregistered [ 2758.775091] Key type ._llcrypt registered [ 2758.777522] Key type .llcrypt registered [ 2758.890546] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2759.798236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2759.904183] alg: No test for adler32 (adler32-zlib) [ 2761.092696] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 2761.305431] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2762.952346] Key type lgssc registered [ 2764.000906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2771.705912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2771.726053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.742557] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2784.651784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2785.360305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2790.369282] LustreError: 86607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2790.382223] LustreError: 86607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2790.540421] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2795.492197] LustreError: 86610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2795.511732] LustreError: 86610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2800.616856] LustreError: 86611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.655189] LustreError: 86611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2800.812779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2801.321198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.758653] LustreError: 86604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2806.784936] LustreError: 86604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2808.064937] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2814.956812] LustreError: 86604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2814.984460] LustreError: 86604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2817.259911] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2817.279491] Lustre: MGS: Client a32a7057-dbc5-4def-be0c-c3c0d9d83ef9 (at 0@lo) reconnecting [ 2817.327868] LustreError: 86621:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2817.532755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2817.633398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2821.435619] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2837.987860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2837.991810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2838.001813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2841.533552] Lustre: server umount lustre-OST0000 complete [ 2848.224602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2848.233289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.241408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2848.253851] Lustre: Skipped 2 previous similar messages [ 2848.260405] Lustre: Skipped 1 previous similar message [ 2851.400389] Lustre: server umount lustre-MDT0000 complete [ 2853.344451] LustreError: 86610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.374178] LustreError: 86610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2856.207582] Lustre: server umount lustre-MDT0001 complete [ 2859.654351] Lustre: server umount MGS complete [ 2870.838353] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:40:35 (1787582435) [ 2901.640951] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 2911.848713] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2952.495382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.001389] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2968.169564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2968.453424] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2968.506041] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2968.597258] Lustre: lustre-MDT0001: new disk, initializing [ 2968.699427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2968.734673] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2968.747522] Lustre: 93794:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2968.753252] LustreError: 93794:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2973.873870] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2986.276952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2986.731457] Lustre: lustre-OST0000: new disk, initializing [ 2986.745549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2986.759366] Lustre: 94561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2986.888317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.411824] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2997.345078] Lustre: 94986:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005bc1da4e x1874415926157696/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787582579 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3005.945446] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3014.416871] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3014.453540] Lustre: MGS: Client 9aae3432-4e0a-4373-9bfd-6d0ddbff5254 (at 0@lo) reconnecting [ 3014.544712] Lustre: 93063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3014.554255] LustreError: 93780:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3014.562736] LustreError: 93780:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3014.607465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3014.736453] Lustre: lustre-MDT0000: new disk, initializing [ 3014.873750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.897406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3014.920715] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3015.969475] Lustre: 95630:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000020050085 x1874415926164608/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787582598 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3020.121255] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3021.041836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3021.052802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3021.165459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3046.371097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3046.372944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3046.377505] Lustre: Skipped 1 previous similar message [ 3046.385155] Lustre: Skipped 1 previous similar message [ 3051.492658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3051.508642] Lustre: Skipped 1 previous similar message [ 3052.462302] Lustre: server umount lustre-OST0000 complete [ 3061.728366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3061.731097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.750323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3061.761040] Lustre: Skipped 1 previous similar message [ 3063.050897] Lustre: server umount lustre-MDT0000 complete [ 3066.851080] LustreError: 93764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.870621] LustreError: 93764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3073.958045] Lustre: server umount lustre-MDT0001 complete [ 3078.259574] Lustre: server umount MGS complete [ 3090.993675] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:44:15 (1787582655) [ 3112.563330] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3115.335277] Key type lgssc unregistered [ 3115.618671] LNet: 98131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3115.627226] LNetError: 98131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3115.645207] LNet: Removed LNI 192.168.201.136@tcp [ 3116.603582] Key type .llcrypt unregistered [ 3116.605564] Key type ._llcrypt unregistered [ 3136.120049] Key type ._llcrypt registered [ 3136.122632] Key type .llcrypt registered [ 3136.287962] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3137.655805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3137.732507] alg: No test for adler32 (adler32-zlib) [ 3138.906700] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 3139.398497] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3141.159753] Key type lgssc registered [ 3142.517562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3153.142649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3153.180480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.855110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3161.245538] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3165.153669] LustreError: 99354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3165.181320] LustreError: 99354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3170.288232] LustreError: 99355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3171.720554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3172.160878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3176.819837] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3186.593568] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3193.200373] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3201.036239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3201.407327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.378514] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3218.208804] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3224.979403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3240.425477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.447667] Lustre: Skipped 1 previous similar message [ 3240.450056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3240.469247] Lustre: Skipped 1 previous similar message [ 3244.789366] Lustre: server umount lustre-OST0000 complete [ 3249.122526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3249.134285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.150571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3250.660344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.662929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3255.365466] Lustre: server umount lustre-MDT0000 complete [ 3255.778785] LustreError: 99350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.806190] LustreError: 99350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3258.925305] LustreError: 99335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582825 with bad export cookie 17045965744781620285 [ 3258.934246] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3259.253468] Lustre: server umount lustre-MDT0001 complete [ 3271.788723] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3274.592484] Key type lgssc unregistered [ 3274.952816] LNet: 102917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3274.958682] LNetError: 102917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3276.013048] LNet: Removed LNI 192.168.201.136@tcp [ 3276.845495] Key type .llcrypt unregistered [ 3276.848764] Key type ._llcrypt unregistered [ 3295.038303] Key type ._llcrypt registered [ 3295.040421] Key type .llcrypt registered [ 3295.116772] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3296.396119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3296.430551] alg: No test for adler32 (adler32-zlib) [ 3297.558571] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 3297.769857] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3299.431552] Key type lgssc registered [ 3300.446538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3306.832467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3306.835230] LustreError: 103999:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3306.842910] Lustre: Skipped 1 previous similar message [ 3306.842931] LustreError: 104000:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3306.863733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.866839] LustreError: 103999:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3306.866870] LustreError: 103999:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3306.866890] LustreError: 103999:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3308.516390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.503564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3316.820742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3321.727275] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3327.517703] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing lsmod [ 3330.261986] LustreError: 105202:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3330.293987] LustreError: 105206:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3330.300927] LustreError: 105202:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4964 [ 3330.307380] LustreError: 105202:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3330.317468] LustreError: 105202:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3330.334763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3330.743246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3332.798147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3337.184460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3337.187691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3337.204231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3338.209871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3338.212570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3338.239652] Lustre: Skipped 1 previous similar message [ 3338.259638] Lustre: Skipped 1 previous similar message [ 3340.208562] Lustre: server umount lustre-MDT0000 complete [ 3342.375270] LustreError: 104038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3342.387303] LustreError: 104038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3343.918108] LustreError: 104017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582910 with bad export cookie 7071892647655617044 [ 3343.922054] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3343.927733] LustreError: 104017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3344.504843] LustreError: 104469:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003d8fb712 x1874416489223424/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3344.711910] Lustre: server umount lustre-MDT0001 complete [ 3359.565567] Lustre: server umount lustre-OST0000 complete [ 3369.270206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.716719] LustreError: 106272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3369.798770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3374.692660] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3375.086253] LustreError: 106273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3380.202292] LustreError: 106272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.313777] LustreError: 106273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.853042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3388.718374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3395.025814] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3405.394693] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3412.710764] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3420.381803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3420.913582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.599207] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3438.709476] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3440.403228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3453.921925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3453.929132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.937073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3453.939669] Lustre: Skipped 1 previous similar message [ 3455.968722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3455.997486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3456.009590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3458.376014] Lustre: server umount lustre-OST0000 complete [ 3465.696046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3465.707773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.714624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3465.718279] Lustre: Skipped 1 previous similar message [ 3468.492745] Lustre: server umount lustre-MDT0000 complete [ 3471.331036] LustreError: 106268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3471.350619] LustreError: 106268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3472.912350] LustreError: 106254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583039 with bad export cookie 7071892647655617989 [ 3472.919293] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3472.924231] LustreError: 106254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3473.653440] Lustre: server umount lustre-MDT0001 complete [ 3485.740209] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3488.575145] Key type lgssc unregistered [ 3488.847616] LNet: 109831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3488.855599] LNetError: 109831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3488.871698] LNet: Removed LNI 192.168.201.136@tcp [ 3489.573839] Key type .llcrypt unregistered [ 3489.576120] Key type ._llcrypt unregistered [ 3514.525207] Key type ._llcrypt registered [ 3514.529679] Key type .llcrypt registered [ 3514.650682] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3516.002131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3516.058530] alg: No test for adler32 (adler32-zlib) [ 3517.061218] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 3517.314280] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3518.983196] Key type lgssc registered [ 3520.137937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3527.148765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3527.175752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3528.573450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3532.535901] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3537.890745] LustreError: 111247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.905354] LustreError: 111247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3540.646700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3540.834818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3544.591197] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3551.603174] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3557.251324] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3563.576771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3563.925465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.097664] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3573.259371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3579.043505] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3584.882659] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3589.134734] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3593.695700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3593.698656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3593.711046] Lustre: Skipped 1 previous similar message [ 3593.712585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3598.841586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3598.853919] Lustre: Skipped 2 previous similar messages [ 3600.333630] Lustre: server umount lustre-OST0000 complete [ 3607.520862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3607.523764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.533630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3609.059514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3611.729709] Lustre: server umount lustre-MDT0000 complete [ 3614.177482] LustreError: 111242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3614.196870] LustreError: 111242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3616.248060] LustreError: 111228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583182 with bad export cookie 13985775010414699210 [ 3616.250854] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3616.265997] LustreError: 111228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3616.996688] Lustre: server umount lustre-MDT0001 complete [ 3630.930866] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3634.433950] Key type lgssc unregistered [ 3634.971232] LNet: 114887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3635.007507] LNetError: 114887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3635.027664] LNet: Removed LNI 192.168.201.136@tcp [ 3636.508825] Key type .llcrypt unregistered [ 3636.511385] Key type ._llcrypt unregistered [ 3661.100977] Key type ._llcrypt registered [ 3661.103721] Key type .llcrypt registered [ 3661.242840] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3662.582478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3662.607424] alg: No test for adler32 (adler32-zlib) [ 3663.759416] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 3664.121022] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3665.871564] Key type lgssc registered [ 3667.460368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3676.507761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3676.538169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.125380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3683.010702] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3688.421827] LustreError: 116110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3688.441911] LustreError: 116110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3692.540505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3693.069388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3698.456456] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3706.655417] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3713.662927] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3721.477653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3721.733724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3722.804173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3729.082182] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3737.950476] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3748.323923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3748.331882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.351068] LustreError: Skipped 1 previous similar message [ 3748.371083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3748.382544] Lustre: Skipped 1 previous similar message [ 3748.397848] Lustre: Skipped 1 previous similar message [ 3753.448938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3753.457741] Lustre: Skipped 1 previous similar message [ 3754.454162] Lustre: server umount lustre-OST0000 complete [ 3763.682074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.694717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3763.704280] Lustre: Skipped 1 previous similar message [ 3763.729959] Lustre: Skipped 1 previous similar message [ 3766.312608] Lustre: server umount lustre-MDT0000 complete [ 3768.808488] LustreError: 116844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.829527] LustreError: 116844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3770.721792] LustreError: 116091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583337 with bad export cookie 4324325126573671202 [ 3770.725856] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3770.735188] LustreError: 116091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3771.147667] Lustre: server umount lustre-MDT0001 complete [ 3782.596796] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3785.610982] Key type lgssc unregistered [ 3785.951678] LNet: 119650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3785.957439] LNetError: 119650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3785.976207] LNet: Removed LNI 192.168.201.136@tcp [ 3786.951183] Key type .llcrypt unregistered [ 3786.953652] Key type ._llcrypt unregistered [ 3812.291981] Key type ._llcrypt registered [ 3812.294269] Key type .llcrypt registered [ 3812.370703] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3813.962750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3813.971804] alg: No test for adler32 (adler32-zlib) [ 3814.998701] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 3815.231870] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3816.943183] Key type lgssc registered [ 3818.273658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3825.740852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3825.768322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3827.350438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3832.724615] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3836.897973] LustreError: 121068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3836.918167] LustreError: 121068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3842.024522] LustreError: 121069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3842.202644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3842.598185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3847.679277] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3855.637620] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3862.464466] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3869.800737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3870.217994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3872.269295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3877.589279] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3886.425053] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3892.875979] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3897.095608] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3897.103292] Lustre: Skipped 2 previous similar messages [ 3900.058421] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3900.106698] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3902.189298] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3902.196613] Lustre: Skipped 2 previous similar messages [ 3908.679043] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3908.687058] Lustre: Skipped 1 previous similar message [ 3914.266186] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.36@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3914.282125] Lustre: Skipped 2 previous similar messages [ 3917.596674] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.36@tcp 192.168.0.[2,10]@tcp [ 3923.513919] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.36/24@tcp in log lustre-MDT0000 [ 3923.521408] Lustre: Skipped 2 previous similar messages [ 3926.758473] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.36/24@tcp [ 3926.768074] Lustre: Skipped 1 previous similar message [ 3933.666305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.670120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3933.695975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3933.724663] LustreError: Skipped 1 previous similar message [ 3938.790185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3938.802597] Lustre: Skipped 2 previous similar messages [ 3939.757748] Lustre: server umount lustre-OST0000 complete [ 3944.938842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3944.951559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3944.976138] Lustre: Skipped 1 previous similar message [ 3944.986443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3949.032058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.035488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3949.049401] Lustre: Skipped 1 previous similar message [ 3950.710456] Lustre: server umount lustre-MDT0000 complete [ 3954.145312] LustreError: 124574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3954.158612] LustreError: 124574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3955.113403] LustreError: 121051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583521 with bad export cookie 8963251366565769008 [ 3955.117697] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3955.128368] LustreError: 121051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3955.376185] Lustre: server umount lustre-MDT0001 complete [ 3967.719993] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3970.460695] Key type lgssc unregistered [ 3970.824497] LNet: 125919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3970.831645] LNetError: 125919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3971.879376] LNet: Removed LNI 192.168.201.136@tcp [ 3972.614408] Key type .llcrypt unregistered [ 3972.618939] Key type ._llcrypt unregistered [ 4005.608755] Key type ._llcrypt registered [ 4005.612331] Key type .llcrypt registered [ 4005.736304] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4007.449228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4007.607073] alg: No test for adler32 (adler32-zlib) [ 4008.700650] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 4009.070359] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 4010.799291] Key type lgssc registered [ 4012.087751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4020.444870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4020.480883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4021.986269] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4022.016274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.018358] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.36/24@tcp [ 4026.708401] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4032.494763] LustreError: 127482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4032.519389] LustreError: 127482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4036.181306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4036.609364] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4036.659073] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.36/24@tcp [ 4036.721162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4041.902557] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4049.833895] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4056.359749] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4064.597587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4065.091819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4067.229749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4072.366653] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4080.614991] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4091.893521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.896140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4091.906334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4091.913595] Lustre: Skipped 1 previous similar message [ 4093.396927] Lustre: server umount lustre-OST0000 complete [ 4098.021548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4098.027722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4098.041214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4098.046633] Lustre: Skipped 1 previous similar message [ 4102.113696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4102.114205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4102.144079] Lustre: Skipped 1 previous similar message [ 4107.232753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4107.240477] Lustre: Skipped 1 previous similar message [ 4112.353922] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4112.356266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4112.375755] Lustre: Skipped 1 previous similar message [ 4112.694830] Lustre: server umount lustre-MDT0000 complete [ 4117.070966] LustreError: 127464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583683 with bad export cookie 9744631883266839942 [ 4117.074895] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4117.092963] LustreError: 127464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4117.484616] Lustre: server umount lustre-MDT0001 complete [ 4129.948761] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 4132.528809] Key type lgssc unregistered [ 4132.809981] LNet: 131085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4132.816681] LNetError: 131085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4132.843767] LNet: Removed LNI 192.168.201.136@tcp [ 4133.850960] Key type .llcrypt unregistered [ 4133.853932] Key type ._llcrypt unregistered [ 4166.548605] Key type ._llcrypt registered [ 4166.554158] Key type .llcrypt registered [ 4166.681875] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 4183.771774] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4185.333180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4185.672267] alg: No test for adler32 (adler32-zlib) [ 4186.820868] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 4187.034149] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 4188.711219] Key type lgssc registered [ 4190.240604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4228.887337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4228.925471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4230.259037] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4230.316104] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4230.501351] Lustre: lustre-MDT0000: new disk, initializing [ 4230.611713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4230.641766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4236.945291] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4249.731769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4249.914677] Lustre: 134489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4249.972726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4249.978426] Lustre: Skipped 1 previous similar message [ 4250.082734] Lustre: lustre-MDT0001: new disk, initializing [ 4250.233835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4250.313805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4250.330252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4255.842932] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4266.175276] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4272.886427] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4280.658329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4281.169960] Lustre: lustre-OST0000: new disk, initializing [ 4281.185377] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4281.209648] Lustre: 136456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4281.327488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.299875] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4289.666445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4289.694595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4289.852242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4298.473359] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4304.098556] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4304.345736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4311.674659] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4312.023279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4315.618146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4315.620610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4315.669492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4315.692878] Lustre: Skipped 1 previous similar message [ 4321.110657] Lustre: server umount lustre-OST0000 complete [ 4325.858820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.877664] Lustre: Skipped 1 previous similar message [ 4325.892951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4325.903328] Lustre: Skipped 1 previous similar message [ 4330.979032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4330.988394] Lustre: Skipped 1 previous similar message [ 4331.780798] Lustre: server umount lustre-MDT0000 complete [ 4335.766823] LustreError: 134481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583902 with bad export cookie 16915611230460097376 [ 4335.771686] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4335.781930] LustreError: 134481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4336.171719] Lustre: server umount lustre-MDT0001 complete [ 4345.255899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4345.668129] LustreError: 138675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4345.687746] LustreError: 138675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4345.813499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4350.729991] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4350.955383] LustreError: 138676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.066233] LustreError: 138675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4358.898061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.196387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4364.086697] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4372.292367] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4378.625513] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4387.230950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4387.734826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4395.273982] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4403.876740] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4408.441665] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4408.598544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4413.030202] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4413.218857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4423.254555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4423.483483] Lustre: lustre-OST0001: new disk, initializing [ 4423.486673] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4423.494295] Lustre: 142045:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4423.631295] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4430.031623] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4432.533909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4432.544536] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4432.651265] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4438.536265] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4438.724757] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4443.278139] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4443.442421] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4449.831505] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a4e05ca7800.ost_server_uuid 50 [ 4451.617294] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a4e05ca7800.ost_server_uuid in FULL state after 0 sec [ 4463.593728] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4463.604558] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4463.619585] Lustre: Skipped 1 previous similar message [ 4463.629959] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4463.651065] Lustre: Skipped 1 previous similar message [ 4468.707469] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4468.723851] Lustre: Skipped 1 previous similar message [ 4476.383188] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4476.696254] Lustre: server umount lustre-OST0001 complete [ 4478.953104] LustreError: 142506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4478.987259] LustreError: 142506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4482.017716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4482.029338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.043699] Lustre: Skipped 1 previous similar message [ 4482.049658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4482.059259] Lustre: Skipped 2 previous similar messages [ 4484.087271] LustreError: 140500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4484.111549] LustreError: 140500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4487.779435] Lustre: server umount lustre-OST0000 complete [ 4494.307600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.321632] Lustre: Skipped 1 previous similar message [ 4498.832445] Lustre: server umount lustre-MDT0000 complete [ 4499.425478] LustreError: 138672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.440353] LustreError: 138672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4503.066317] LustreError: 138656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584069 with bad export cookie 16915611230460098265 [ 4503.070613] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4503.079891] LustreError: 138656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4503.576739] Lustre: server umount lustre-MDT0001 complete [ 4527.666427] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 4530.500649] Key type lgssc unregistered [ 4530.815659] LNet: 145055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4530.827368] LNetError: 145055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4530.850427] LNet: Removed LNI 192.168.201.136@tcp [ 4531.915318] Key type .llcrypt unregistered [ 4531.918348] Key type ._llcrypt unregistered [ 4578.044667] Key type ._llcrypt registered [ 4578.047440] Key type .llcrypt registered [ 4578.161465] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 4596.369409] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4597.628486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4597.707345] alg: No test for adler32 (adler32-zlib) [ 4599.331231] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 4599.719256] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 4601.423198] Key type lgssc registered [ 4603.248434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4648.082452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4648.146632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4649.395923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4649.427641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4649.511730] Lustre: lustre-MDT0000: new disk, initializing [ 4649.589649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.639284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4654.373365] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4665.379935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4665.495965] Lustre: 149217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4665.552334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4665.556880] Lustre: Skipped 1 previous similar message [ 4665.636489] Lustre: lustre-MDT0001: new disk, initializing [ 4665.722538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4665.782201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4665.793184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4670.714362] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4681.626596] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4688.526850] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4697.381042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4697.767558] Lustre: lustre-OST0000: new disk, initializing [ 4697.770509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4697.779799] Lustre: 151184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4697.867924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4703.493900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4703.499977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4703.520470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 4705.074912] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4720.418725] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4727.479922] Lustre: Failing over lustre-OST0000 [ 4727.638719] Lustre: server umount lustre-OST0000 complete [ 4744.671990] Lustre: 147438:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584295/real 1787584295] req@000000009b43f4d2 x1874417854484992/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787584311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4744.733479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4745.696108] Lustre: 147435:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584295/real 1787584295] req@00000000999c5c57 x1874417854485248/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787584311 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4745.725116] Lustre: 147435:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4749.787207] Lustre: 147436:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584300/real 1787584300] req@00000000e067246c x1874417854486400/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787584316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4749.832477] Lustre: 147436:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4751.189713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4751.767695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4751.817619] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4754.638340] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4754.783586] Lustre: 147438:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584305/real 1787584305] req@0000000036056b48 x1874417854487552/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787584321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4754.940226] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4754.940350] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4761.058292] Lustre: 147437:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584311/real 1787584311] req@0000000095c25c52 x1874417854488576/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787584327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4761.126729] Lustre: 147437:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4762.874960] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4768.117956] Lustre: Failing over lustre-MDT0000 [ 4768.231348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4768.236363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4768.247626] Lustre: Skipped 1 previous similar message [ 4768.385303] Lustre: server umount lustre-MDT0000 complete [ 4770.286685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4770.313542] Lustre: Skipped 1 previous similar message [ 4775.110764] LustreError: 149222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4775.154270] LustreError: 149222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4780.237086] LustreError: 149221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4780.292627] LustreError: 149221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4785.355129] LustreError: 149222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4785.378457] LustreError: 149222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4786.662737] Lustre: 147435:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584336/real 1787584336] req@00000000ca07e57e x1874417854500352/t0(0) o400->MGC192.168.201.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787584352 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4786.714544] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4790.884084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4791.497657] LustreError: 149223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4791.517696] LustreError: 149223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4795.596762] LustreError: 149223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4795.625878] LustreError: 149223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4796.910588] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa407fca5ae26fcff [ 4796.923876] Lustre: MGC192.168.201.136@tcp: Connection restored to 0@lo (at 0@lo) [ 4796.933188] Lustre: Skipped 1 previous similar message [ 4797.159411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4797.203634] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4800.720114] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4801.949787] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4802.548406] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4802.561529] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4802.627428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 4811.237766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4811.244505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4811.259359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4812.777615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4812.783878] Lustre: Skipped 1 previous similar message [ 4816.873898] Lustre: server umount lustre-OST0000 complete [ 4823.009760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4823.011280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4823.029146] Lustre: Skipped 1 previous similar message [ 4823.030128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4827.829783] Lustre: server umount lustre-MDT0000 complete [ 4828.128580] LustreError: 150099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4828.162919] LustreError: 150099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4832.121376] LustreError: 149207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584398 with bad export cookie 11819693535578750207 [ 4832.143205] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4832.152124] LustreError: 149207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4832.504925] Lustre: server umount lustre-MDT0001 complete [ 4842.927920] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 4845.537662] Key type lgssc unregistered [ 4845.817770] LNet: 154848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4845.823248] LNetError: 154848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4846.897875] LNet: Removed LNI 192.168.201.136@tcp [ 4847.770107] Key type .llcrypt unregistered [ 4847.772546] Key type ._llcrypt unregistered [ 4885.146614] Key type ._llcrypt registered [ 4885.149335] Key type .llcrypt registered [ 4885.243585] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4886.772652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4886.799790] alg: No test for adler32 (adler32-zlib) [ 4887.936375] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 4888.269972] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 4890.000897] Key type lgssc registered [ 4891.937342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4901.252784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4901.271021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4902.884961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4907.505455] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4913.141645] LustreError: 156370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.163850] LustreError: 156370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4917.897476] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4918.241033] LustreError: 156371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.069557] LustreError: 156400:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4922.076477] LustreError: 156400:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4922.086900] LustreError: 156400:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4922.548813] Lustre: server umount lustre-MDT0000 complete [ 4940.347260] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 4951.175494] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4993.792887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4994.033125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4994.051332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4994.114831] Lustre: lustre-MDT0000: new disk, initializing [ 4994.192745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4994.215141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4998.492750] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5008.349203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5008.467215] Lustre: 159716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5008.581255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5008.588257] Lustre: Skipped 1 previous similar message [ 5008.715406] Lustre: lustre-MDT0001: new disk, initializing [ 5008.807445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5008.855451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5008.875637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5013.265492] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5022.488572] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5028.848421] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5035.742891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5036.021276] Lustre: lustre-OST0000: new disk, initializing [ 5036.025421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5036.030741] Lustre: 161682:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5036.101360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5042.705605] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5042.749542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5042.755846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5042.933537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5052.436471] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5057.875677] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5059.887471] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:17:04 (1787584624) [ 5335.008389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5335.010498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5335.031266] Lustre: Skipped 2 previous similar messages [ 5335.041954] Lustre: Skipped 2 previous similar messages [ 5338.807368] Lustre: server umount lustre-MDT0000 complete [ 5340.128831] LustreError: 159727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5340.158425] LustreError: 159727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5342.872688] LustreError: 159707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584909 with bad export cookie 8940529928477678687 [ 5342.873905] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5342.886336] LustreError: 159707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5345.251691] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5345.255380] LustreError: 159722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5345.268071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5347.298793] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5349.250455] Lustre: server umount lustre-MDT0001 complete [ 5360.455623] Lustre: server umount lustre-OST0000 complete [ 5371.543819] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 5384.390983] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5427.426382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5427.627988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5427.651347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5427.714442] Lustre: lustre-MDT0000: new disk, initializing [ 5427.781799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5427.798095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5432.878742] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5444.977125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5445.170854] Lustre: 166353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5445.234773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5445.241708] Lustre: Skipped 1 previous similar message [ 5445.333503] Lustre: lustre-MDT0001: new disk, initializing [ 5445.430493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5445.474179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5445.490771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5450.484944] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5461.136512] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5468.174851] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5476.121372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5476.386223] Lustre: lustre-OST0000: new disk, initializing [ 5476.389236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5476.400820] Lustre: 168322:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5476.511934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5483.558773] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5484.128632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5484.136842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5484.261519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5494.591890] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5499.797072] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5500.013303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5504.834346] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5505.108522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5510.112718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5510.113751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5510.118475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5510.121799] LustreError: Skipped 1 previous similar message [ 5510.145844] Lustre: Skipped 1 previous similar message [ 5513.451752] Lustre: server umount lustre-OST0000 complete [ 5517.280346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5517.292619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5517.306388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5517.308992] Lustre: Skipped 1 previous similar message [ 5523.833823] Lustre: server umount lustre-MDT0000 complete [ 5525.472873] LustreError: 166360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.492955] LustreError: 166360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5528.175049] LustreError: 166345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585094 with bad export cookie 8940529928477837517 [ 5528.181113] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5528.749249] Lustre: server umount lustre-MDT0001 complete [ 5539.880473] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:25:04 (1787585104) [ 5560.205814] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 5577.554307] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5620.770149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5621.191368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5621.250341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5621.349322] Lustre: lustre-MDT0000: new disk, initializing [ 5621.421273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5621.442374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5626.668868] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5640.797064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.952101] Lustre: 173102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5641.166166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5647.734958] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5658.892694] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5665.699341] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5673.946033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5674.256585] Lustre: lustre-OST0000: new disk, initializing [ 5674.260146] Lustre: Skipped 1 previous similar message [ 5674.271691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5674.283699] Lustre: Skipped 2 previous similar messages [ 5674.289215] Lustre: 175078:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5680.748897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5680.763673] Lustre: Skipped 1 previous similar message [ 5680.776180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5680.931508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5681.089039] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5692.289238] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5701.618882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.622187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5701.633085] Lustre: Skipped 1 previous similar message [ 5701.646446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5701.646761] LustreError: Skipped 1 previous similar message [ 5701.684554] Lustre: Skipped 3 previous similar messages [ 5706.763718] Lustre: server umount lustre-OST0000 complete [ 5711.843317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.863241] Lustre: Skipped 2 previous similar messages [ 5718.156678] Lustre: server umount lustre-MDT0000 complete [ 5722.094882] LustreError: 175749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5722.128720] LustreError: 175749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5723.023835] LustreError: 173094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585289 with bad export cookie 8940529928477838392 [ 5723.030861] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5723.044857] LustreError: 173094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5723.742277] Lustre: server umount lustre-MDT0001 complete [ 5736.084523] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:28:20 (1787585300) [ 5757.521453] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 5770.285485] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5807.989968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.339819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5808.387768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5808.484980] Lustre: lustre-MDT0000: new disk, initializing [ 5808.595294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5808.603679] Lustre: Skipped 2 previous similar messages [ 5808.626067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5813.570965] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5824.649170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5824.824954] Lustre: 179401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5825.177125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5830.257877] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5841.443552] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5848.781089] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5856.481994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5856.807317] Lustre: 181379:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5862.975349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5863.100086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5864.092661] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5874.644463] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5883.874387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5883.895153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.896832] LustreError: Skipped 1 previous similar message [ 5883.900078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5883.922770] Lustre: Skipped 1 previous similar message [ 5883.945732] Lustre: Skipped 5 previous similar messages [ 5888.402601] Lustre: server umount lustre-OST0000 complete [ 5898.806562] Lustre: server umount lustre-MDT0000 complete [ 5899.253205] LustreError: 179408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5899.296710] LustreError: 179408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5902.924351] LustreError: 179393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585469 with bad export cookie 8940529928477839484 [ 5902.934851] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5902.937867] LustreError: 179393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5903.213568] Lustre: server umount lustre-MDT0001 complete [ 5920.482308] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 5924.713273] Key type lgssc unregistered [ 5925.031205] LNet: 183240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5925.036824] LNetError: 183240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5926.056691] LNet: Removed LNI 192.168.201.136@tcp [ 5926.957714] Key type .llcrypt unregistered [ 5926.969023] Key type ._llcrypt unregistered [ 5954.642609] Key type ._llcrypt registered [ 5954.645765] Key type .llcrypt registered [ 5954.782384] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5955.942796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5956.047100] alg: No test for adler32 (adler32-zlib) [ 5957.342534] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 5957.718427] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 5959.479182] Key type lgssc registered [ 5961.182880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5971.484734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5971.508398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5973.109755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5978.418378] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5982.702255] LustreError: 184657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5982.726440] LustreError: 184657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5987.812347] LustreError: 184656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5990.107346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5990.818395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5996.736056] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6006.208653] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6012.493842] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6019.637734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6020.006636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6026.640072] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6035.854915] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6043.599425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6069.227593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6069.240956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.242904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6069.257763] Lustre: Skipped 1 previous similar message [ 6069.276778] Lustre: Skipped 1 previous similar message [ 6072.485750] Lustre: server umount lustre-OST0000 complete [ 6077.921434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6077.937187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6077.961148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6079.458974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6079.459630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6084.577401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6084.590218] Lustre: Skipped 2 previous similar messages [ 6088.675826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6088.684302] Lustre: Skipped 1 previous similar message [ 6091.231134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6091.398528] Lustre: server umount lustre-MDT0000 complete [ 6093.794477] LustreError: 186274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6093.802886] LustreError: 186274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6095.032060] LustreError: 184638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585661 with bad export cookie 942440790059950597 [ 6095.039684] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6095.044191] LustreError: 184638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6095.465154] Lustre: server umount lustre-MDT0001 complete [ 6106.735159] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 6110.107318] Key type lgssc unregistered [ 6110.448252] LNet: 188220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6110.454255] LNetError: 188220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6110.473112] LNet: Removed LNI 192.168.201.136@tcp [ 6111.411890] Key type .llcrypt unregistered [ 6111.415882] Key type ._llcrypt unregistered [ 6144.004514] Key type ._llcrypt registered [ 6144.006570] Key type .llcrypt registered [ 6144.115458] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 6146.094964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6146.243196] alg: No test for adler32 (adler32-zlib) [ 6147.596990] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 6148.012409] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 6149.736094] Key type lgssc registered [ 6150.985187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6160.964741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6160.975317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6162.602116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6168.220889] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6172.132877] LustreError: 189638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6172.175466] LustreError: 189638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6177.252261] LustreError: 189637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6179.267363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6179.817443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6184.933654] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6194.619932] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6201.692834] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6212.189807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6212.800293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6220.704060] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6229.055674] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6232.307949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6240.737129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6240.746966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6240.759575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6242.784908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6242.787366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6242.826414] Lustre: Skipped 2 previous similar messages [ 6246.099580] Lustre: lustre-OST0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 6247.070694] Lustre: server umount lustre-OST0000 complete [ 6254.045751] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8a4e0876d000.ost_server_uuid 50 [ 6255.562317] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6271.968783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6271.986234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6272.010323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6276.158561] Lustre: server umount lustre-MDT0000 complete [ 6278.624425] LustreError: 190620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6278.649455] LustreError: 190620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6280.419682] LustreError: 189619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585846 with bad export cookie 18263892109582452724 [ 6280.427936] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6280.434931] LustreError: 189619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6280.715205] Lustre: server umount lustre-MDT0001 complete [ 6290.172050] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:37:34 (1787585854) [ 6300.072199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6300.980989] LustreError: 193557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6301.099875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6306.283626] LustreError: 193558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6306.329485] LustreError: 193558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6309.180528] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6316.527791] LustreError: 193557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.556974] LustreError: 193557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6319.284754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6319.864720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6326.024848] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6334.705083] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6341.147615] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6350.180374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6350.617035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6358.266311] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6370.088764] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6371.151758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6379.086574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6379.363402] Lustre: lustre-OST0001: new disk, initializing [ 6379.368499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6379.376444] Lustre: 196402:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6379.567803] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6387.847695] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6388.402753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6388.425567] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6388.586134] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6398.685528] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6402.018555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6402.024401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.040253] Lustre: Skipped 1 previous similar message [ 6402.045222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6402.051953] Lustre: Skipped 2 previous similar messages [ 6414.305865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6414.316650] Lustre: Skipped 7 previous similar messages [ 6416.351407] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6416.522763] Lustre: server umount lustre-OST0000 complete [ 6419.436327] LustreError: 195381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6419.453139] LustreError: 195381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6424.202468] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6424.470677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6429.045966] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6429.277767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6445.026646] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6445.030875] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6445.035850] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6445.041725] Lustre: Skipped 2 previous similar messages [ 6445.051095] Lustre: Skipped 2 previous similar messages [ 6450.057946] Lustre: server umount lustre-OST0001 complete [ 6455.266750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6469.087209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6469.308718] Lustre: server umount lustre-MDT0000 complete [ 6470.629157] LustreError: 194286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6470.648812] LustreError: 194286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 6472.637772] LustreError: 193537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586038 with bad export cookie 18263892109582453795 [ 6472.642189] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6472.648558] LustreError: 193537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6472.880445] Lustre: server umount lustre-MDT0001 complete [ 6497.664870] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:41:02 (1787586062) [ 6504.682958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6505.059966] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6505.090623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6505.303132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6510.281669] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6518.403251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6518.479281] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6518.518494] Lustre: 199845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6522.741930] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6529.371402] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6535.535699] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6541.981694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6542.127397] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6548.919112] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6549.647058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6556.130136] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6563.290673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6563.396506] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6571.293610] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6582.077906] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6585.319131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6585.327069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6585.340503] Lustre: Skipped 1 previous similar message [ 6585.357625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6585.371710] Lustre: Skipped 6 previous similar messages [ 6591.330164] Lustre: server umount lustre-OST0000 complete [ 6593.227521] LustreError: 202333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6609.389367] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6609.405976] LustreError: Skipped 1 previous similar message [ 6612.872385] Lustre: server umount lustre-OST0001 complete [ 6619.618637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6619.634602] Lustre: Skipped 4 previous similar messages [ 6631.391141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6631.608913] Lustre: server umount lustre-MDT0000 complete [ 6635.657831] LustreError: 199836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586201 with bad export cookie 18263892109582455202 [ 6635.661546] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6635.677877] LustreError: 199836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6635.968866] Lustre: server umount lustre-MDT0001 complete [ 6661.354353] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:43:45 (1787586225) [ 6679.082246] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 6689.625223] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 6727.008209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6727.278823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6727.313550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6727.428992] Lustre: lustre-MDT0000: new disk, initializing [ 6727.534336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6727.537546] Lustre: Skipped 3 previous similar messages [ 6727.563421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6732.182871] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6745.353990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6745.541642] Lustre: 207946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6745.598116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6745.609554] Lustre: Skipped 1 previous similar message [ 6745.752495] Lustre: lustre-MDT0001: new disk, initializing [ 6745.899946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6745.906957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6750.783588] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6761.361710] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6768.102485] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6775.320432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6775.548695] Lustre: lustre-OST0000: new disk, initializing [ 6775.551993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6775.557173] Lustre: 209915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6781.489267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6781.502225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6781.582214] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6781.861237] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6792.228310] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6797.032188] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6797.302193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6801.930790] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6802.179345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6807.521098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6807.532221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6807.540028] LustreError: Skipped 1 previous similar message [ 6807.543290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6807.555513] Lustre: Skipped 1 previous similar message [ 6807.581685] Lustre: Skipped 13 previous similar messages [ 6810.453808] Lustre: server umount lustre-OST0000 complete [ 6821.398130] Lustre: server umount lustre-MDT0000 complete [ 6822.883102] LustreError: 207958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6822.930597] LustreError: 207958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6825.612593] LustreError: 207939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586391 with bad export cookie 18263892109582456763 [ 6825.641094] LustreError: 207939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6825.641978] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6825.999278] Lustre: server umount lustre-MDT0001 complete [ 6838.000569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6844.775660] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6856.156169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6856.550811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6856.555953] Lustre: Skipped 3 previous similar messages [ 6862.207059] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6870.535308] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6877.355616] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6886.532584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6895.094833] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6904.274974] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6909.707983] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6910.116197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6914.957214] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6915.272052] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6919.149911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6919.160266] LustreError: Skipped 1 previous similar message [ 6923.992718] Lustre: server umount lustre-OST0000 complete [ 6932.093298] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6940.269776] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6971.363171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6979.499440] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6984.177178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6984.196992] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6984.199426] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6988.107721] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6989.808922] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6999.528664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6999.533085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6999.558075] Lustre: Skipped 4 previous similar messages [ 6999.569244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6999.577406] Lustre: Skipped 4 previous similar messages [ 7001.564892] Lustre: server umount lustre-OST0000 complete [ 7012.350435] Lustre: server umount lustre-MDT0000 complete [ 7016.744173] LustreError: 212113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586583 with bad export cookie 18263892109582457645 [ 7016.756442] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7016.760342] LustreError: 212113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7017.057532] Lustre: server umount lustre-MDT0001 complete [ 7026.368933] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:49:51 (1787586591) [ 7034.156828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.482958] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7049.536894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7054.384855] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7063.059226] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7070.065110] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7078.175406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7085.553660] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7093.461705] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7099.559837] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7099.780940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7104.182749] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7104.395324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7111.701400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7111.960943] Lustre: lustre-OST0001: new disk, initializing [ 7111.964828] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7111.969731] Lustre: 221474:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7118.884554] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7118.956617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7118.967438] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7119.059026] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7130.668482] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7136.409247] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7136.781571] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7142.212423] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7142.397624] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7144.955946] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7144.958078] LustreError: Skipped 1 previous similar message [ 7150.847730] Lustre: server umount lustre-OST0001 complete [ 7155.178115] LustreError: 219927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.215976] LustreError: 219927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7160.533447] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7160.810158] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7166.163711] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7166.487670] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7196.477188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7196.723269] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7196.726344] Lustre: Skipped 6 previous similar messages [ 7201.783364] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7201.802188] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7201.803248] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7201.837956] Lustre: Skipped 2 previous similar messages [ 7204.088392] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7213.440768] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7215.446196] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7223.697787] Lustre: server umount lustre-OST0001 complete [ 7246.656830] Lustre: server umount lustre-OST0000 complete [ 7257.321772] Lustre: server umount lustre-MDT0000 complete [ 7261.730383] LustreError: 218084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586828 with bad export cookie 18263892109582458604 [ 7261.747349] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7261.756142] LustreError: 218084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7268.120629] Lustre: server umount lustre-MDT0001 complete [ 7293.585427] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:54:18 (1787586858) [ 7300.371554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7300.800955] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7300.843371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7306.872861] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7316.476394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7316.553292] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7316.575565] Lustre: 226721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7322.042881] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7332.242320] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7339.124961] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7347.471732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7347.752248] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7354.533570] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7364.521066] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7374.632982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7374.733622] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7382.344761] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7391.658782] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7393.844674] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7399.278470] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7399.535894] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7404.430391] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7404.661898] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7410.873306] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a4e05c37000.ost_server_uuid 50 [ 7413.071582] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a4e05c37000.ost_server_uuid in IDLE state after 0 sec [ 7414.441245] Lustre: Permanently deactivating lustre-OST0001 [ 7414.465786] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7417.913210] Lustre: Permanently reactivating lustre-OST0001 [ 7426.633907] Lustre: server umount lustre-OST0001 complete [ 7428.067234] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7428.071716] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7428.074201] LustreError: Skipped 3 previous similar messages [ 7428.117065] Lustre: Skipped 12 previous similar messages [ 7433.199235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7433.217880] Lustre: Skipped 14 previous similar messages [ 7437.558244] Lustre: server umount lustre-OST0000 complete [ 7448.566598] Lustre: server umount lustre-MDT0000 complete [ 7452.898780] LustreError: 226715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587019 with bad export cookie 18263892109582459899 [ 7452.902617] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7452.917885] LustreError: 226715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7453.826804] Lustre: server umount lustre-MDT0001 complete [ 7482.114956] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:57:26 (1787587046) [ 7490.483334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7490.748887] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7490.798519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7490.809947] Lustre: Skipped 5 previous similar messages [ 7496.276875] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7505.541899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.615748] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7505.638894] Lustre: 233420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7512.283933] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7522.384584] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7529.554391] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7537.060291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7537.285416] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7537.312399] Lustre: Permanently deactivating lustre-OST0000 [ 7537.326645] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7543.876871] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7543.974056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7543.975868] LustreError: 235596:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 7544.695516] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7553.598314] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7562.146678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7562.249777] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7570.133867] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7570.421160] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7579.661659] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7582.013842] Lustre: Permanently reactivating lustre-OST0000 [ 7582.046099] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7582.053637] Lustre: Skipped 2 previous similar messages [ 7589.541580] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7589.591575] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7589.633732] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7589.721340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7589.753851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7601.869679] Lustre: server umount lustre-OST0001 complete [ 7612.307467] Lustre: server umount lustre-OST0000 complete [ 7622.600732] Lustre: server umount lustre-MDT0000 complete [ 7626.513944] LustreError: 237017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587192 with bad export cookie 18263892109582461579 [ 7626.517076] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7626.522361] LustreError: 237017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7626.829806] Lustre: server umount lustre-MDT0001 complete [ 7641.912731] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 7645.009194] Key type lgssc unregistered [ 7645.352646] LNet: 238312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7645.360636] LNetError: 238312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7645.382683] LNet: Removed LNI 192.168.201.136@tcp [ 7646.191543] Key type .llcrypt unregistered [ 7646.193305] Key type ._llcrypt unregistered [ 7674.301195] Key type ._llcrypt registered [ 7674.303523] Key type .llcrypt registered [ 7674.407570] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7675.869307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7675.919154] alg: No test for adler32 (adler32-zlib) [ 7677.093955] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 7677.493021] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 7679.168092] Key type lgssc registered [ 7680.503756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7691.375545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7691.406104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7692.772240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7697.362220] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7703.012795] LustreError: 239781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7703.031184] LustreError: 239781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7706.517238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7706.692892] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7706.753414] Lustre: 239770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7706.763275] Lustre: Permanently deactivating lustre-MDT0001 [ 7706.774768] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7707.132361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7707.524270] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7707.534453] LustreError: 239812:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7707.563669] LustreError: 239812:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7707.580767] LustreError: 239812:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 1, failed: rc = -108 [ 7712.179264] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7720.811111] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7727.149919] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7734.616705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7734.960642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7738.022892] LustreError: 241634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7738.039728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7742.279570] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7743.467928] LustreError: 241811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7748.582130] LustreError: 241634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7751.268594] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7752.684386] LustreError: 241633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7752.725026] LustreError: 241633:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7759.462392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7759.954353] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7765.009878] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7765.010863] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7768.396369] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7777.976161] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7781.834921] Lustre: Permanently reactivating lustre-MDT0001 [ 7781.864413] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7781.870648] Lustre: Skipped 2 previous similar messages [ 7787.268529] Lustre: lustre-MDT0001: Client cc957faa-0058-448e-9860-05b3f2ae4d6a (at 192.168.201.36@tcp) reconnecting [ 7795.141937] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7797.400107] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7803.640824] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7803.849199] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7808.888040] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7809.154865] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7812.980104] Lustre: Permanently deactivating lustre-MDT0001 [ 7813.032448] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7813.037760] Lustre: Skipped 2 previous similar messages [ 7820.379988] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7824.865391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7824.873084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7824.900388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7828.456819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7828.465054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7828.473370] Lustre: Skipped 1 previous similar message [ 7828.486558] Lustre: Skipped 3 previous similar messages [ 7830.741572] Lustre: server umount lustre-MDT0000 complete [ 7833.572580] LustreError: 239781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7833.606987] LustreError: 239781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7835.309734] LustreError: 240544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587401 with bad export cookie 1567376715913893100 [ 7835.318089] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7835.331421] LustreError: 240544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7835.699767] Lustre: server umount lustre-MDT0001 complete [ 7842.034833] Lustre: server umount lustre-OST0000 complete [ 7856.353069] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:03:40 (1787587420) [ 7869.097056] Lustre: server umount lustre-OST0001 complete [ 7878.514587] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 7890.960526] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7933.201691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7933.467261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7933.475450] Lustre: Skipped 2 previous similar messages [ 7933.493710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7933.576213] Lustre: lustre-MDT0000: new disk, initializing [ 7933.663388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7933.691111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7938.437640] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7951.160300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7951.414677] Lustre: 248056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7951.503952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7951.508612] Lustre: Skipped 1 previous similar message [ 7951.616483] Lustre: lustre-MDT0001: new disk, initializing [ 7951.847712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7951.926720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7951.943989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7958.185473] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7970.046551] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7977.524615] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7985.604027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7986.049968] Lustre: lustre-OST0000: new disk, initializing [ 7986.055100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7986.063541] Lustre: 250027:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7986.150962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7993.505867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7993.522437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7993.679714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7994.222110] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8006.389293] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8009.209963] LustreError: 248061:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8018.935071] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8019.159954] Lustre: lustre-OST0001: new disk, initializing [ 8019.164487] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8019.170854] Lustre: 251145:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8019.271925] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8019.297919] LustreError: 248061:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8019.383226] LustreError: 249251:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8027.468336] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8027.736065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8027.755707] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8027.886086] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8029.482699] LustreError: 249251:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8037.919404] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8039.609565] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8043.494142] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8043.503404] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8043.510696] Lustre: Skipped 1 previous similar message [ 8043.519710] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8043.538859] Lustre: Skipped 1 previous similar message [ 8048.619420] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8048.626106] Lustre: Skipped 1 previous similar message [ 8053.741032] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8053.751115] Lustre: Skipped 1 previous similar message [ 8056.293471] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8056.644994] Lustre: server umount lustre-OST0001 complete [ 8058.866627] LustreError: 250011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8058.889894] LustreError: 250011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8075.753622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8075.753775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8075.761594] LustreError: Skipped 1 previous similar message [ 8075.763654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8075.778086] Lustre: Skipped 2 previous similar messages [ 8081.239820] Lustre: server umount lustre-OST0000 complete [ 8084.962949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8084.969772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8090.981747] Lustre: server umount lustre-MDT0000 complete [ 8094.543843] LustreError: 248047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587660 with bad export cookie 1567376715913895697 [ 8094.548492] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8094.550645] LustreError: 248047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8100.969805] Lustre: server umount lustre-MDT0001 complete [ 8113.124609] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8115.783323] Key type lgssc unregistered [ 8116.148832] LNet: 253181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8116.156960] LNetError: 253181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8116.182482] LNet: Removed LNI 192.168.201.136@tcp [ 8117.043419] Key type .llcrypt unregistered [ 8117.044990] Key type ._llcrypt unregistered [ 8158.931188] Key type ._llcrypt registered [ 8158.933034] Key type .llcrypt registered [ 8159.050679] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8160.021174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8160.082078] alg: No test for adler32 (adler32-zlib) [ 8161.295824] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 8161.528081] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8163.135990] Key type lgssc registered [ 8164.158771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8172.869472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8172.892609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8174.223805] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8174.267415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8174.387295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8178.989846] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8188.830565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8189.028848] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8189.122322] Lustre: 255394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8189.500798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8195.813889] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8203.806908] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8209.653752] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8218.122446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8218.374224] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8218.615594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8226.006867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8226.643379] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8234.714814] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8287.716103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8287.723440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8287.733149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8292.320547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8292.326934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8292.340480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8293.757708] Lustre: server umount lustre-OST0000 complete [ 8298.776652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8312.537490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8313.078374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8314.115713] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8314.129024] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8318.276186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8318.309817] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8319.706750] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8328.623543] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8370.657984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8370.671941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.687268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8370.695557] Lustre: Skipped 1 previous similar message [ 8374.756312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.760972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8374.786252] Lustre: Skipped 1 previous similar message [ 8376.775694] Lustre: server umount lustre-OST0000 complete [ 8383.968440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8383.973697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8383.986084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8387.124470] Lustre: server umount lustre-MDT0000 complete [ 8390.122284] LustreError: 257991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8390.141832] LustreError: 257991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8390.947434] LustreError: 255386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587957 with bad export cookie 13911934505777141186 [ 8390.960444] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8390.961471] LustreError: 255386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8391.395343] Lustre: server umount lustre-MDT0001 complete [ 8403.725197] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8406.886930] Key type lgssc unregistered [ 8407.151752] LNet: 260820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8407.163532] LNetError: 260820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8407.178692] LNet: Removed LNI 192.168.201.136@tcp [ 8407.879164] Key type .llcrypt unregistered [ 8407.884155] Key type ._llcrypt unregistered [ 8434.959069] Key type ._llcrypt registered [ 8434.965260] Key type .llcrypt registered [ 8435.113762] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8436.145042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8436.174949] alg: No test for adler32 (adler32-zlib) [ 8437.418497] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 8437.753184] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8439.480317] Key type lgssc registered [ 8441.000889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8449.795253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8449.826895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8451.528689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8456.597067] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8460.770633] LustreError: 262237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8460.790708] LustreError: 262237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8465.890456] LustreError: 262236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8466.103345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.553343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8472.296280] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8481.193678] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8487.937186] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8494.511645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8495.101690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8496.128184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8502.119622] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8510.879463] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8537.068598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8537.075479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8537.085674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8537.098140] Lustre: Skipped 1 previous similar message [ 8540.405949] Lustre: server umount lustre-OST0000 complete [ 8547.298422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8547.322450] Lustre: Skipped 1 previous similar message [ 8547.339336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8547.347437] Lustre: Skipped 1 previous similar message [ 8551.504708] Lustre: server umount lustre-MDT0000 complete [ 8552.420606] LustreError: 262237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8552.439890] LustreError: 262237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8555.741986] LustreError: 262217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588122 with bad export cookie 4250632959058481469 [ 8555.751418] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8555.758743] LustreError: 262217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8556.199897] Lustre: server umount lustre-MDT0001 complete [ 8573.511685] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8577.127488] Key type lgssc unregistered [ 8577.382819] LNet: 266571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8577.386830] LNetError: 266571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8577.404152] LNet: Removed LNI 192.168.201.136@tcp [ 8578.745741] Key type .llcrypt unregistered [ 8578.748933] Key type ._llcrypt unregistered [ 8602.404188] Key type ._llcrypt registered [ 8602.407688] Key type .llcrypt registered [ 8602.558952] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8604.783151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8604.869241] alg: No test for adler32 (adler32-zlib) [ 8606.008462] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 8606.303888] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8608.023222] Key type lgssc registered [ 8609.354081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8617.581345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8617.610143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8619.152878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8624.081226] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8628.714274] LustreError: 267794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8628.726791] LustreError: 267794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8633.828536] LustreError: 267793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8634.976208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8635.591588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8642.333620] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8651.369234] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8657.664347] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8664.577657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8665.054684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8671.209304] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8679.308808] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8689.667294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8706.021523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8706.057496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8706.118183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8710.123198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8710.124951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8710.131921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8711.521808] Lustre: server umount lustre-OST0000 complete [ 8717.792958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8717.802793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8717.833695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8717.844886] Lustre: Skipped 1 previous similar message [ 8720.357630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8720.359013] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8722.380531] Lustre: server umount lustre-MDT0000 complete [ 8725.476274] LustreError: 270000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8725.509211] LustreError: 270000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8726.861107] LustreError: 267774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588293 with bad export cookie 3428498797404601799 [ 8726.869603] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8726.872976] LustreError: 267774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8727.375745] Lustre: server umount lustre-MDT0001 complete [ 8741.125571] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8744.170228] Key type lgssc unregistered [ 8744.520156] LNet: 271497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8744.533326] LNetError: 271497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8744.572839] LNet: Removed LNI 192.168.201.136@tcp [ 8745.506104] Key type .llcrypt unregistered [ 8745.513320] Key type ._llcrypt unregistered [ 8771.217915] Key type ._llcrypt registered [ 8771.220724] Key type .llcrypt registered [ 8771.327482] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8772.304786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8772.406310] alg: No test for adler32 (adler32-zlib) [ 8773.428861] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 8773.615464] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8775.295481] Key type lgssc registered [ 8776.538042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8784.241122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8784.258978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8785.612331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8789.940634] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8795.107545] LustreError: 272915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8795.122217] LustreError: 272915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8798.284081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8798.675354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8803.263313] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8811.822143] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8818.971353] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8828.378967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8828.888584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8829.935150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8835.222565] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8843.944971] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8868.834504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8868.841146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8868.847895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8868.866495] Lustre: Skipped 1 previous similar message [ 8873.822677] Lustre: server umount lustre-OST0000 complete [ 8879.072798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8879.078621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8879.092593] Lustre: Skipped 1 previous similar message [ 8879.103809] Lustre: Skipped 2 previous similar messages [ 8884.193450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8884.206733] Lustre: Skipped 1 previous similar message [ 8889.312574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8889.318370] Lustre: Skipped 1 previous similar message [ 8892.383462] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8892.575474] Lustre: server umount lustre-MDT0000 complete [ 8894.439531] LustreError: 273643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8894.469793] LustreError: 273643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8896.800492] LustreError: 274760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588463 with bad export cookie 4090048409934245234 [ 8896.809459] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8896.809431] LustreError: 274760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8897.307095] Lustre: server umount lustre-MDT0001 complete [ 8911.808229] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8914.378342] Key type lgssc unregistered [ 8914.747252] LNet: 277228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8914.752917] LNetError: 277228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8914.775174] LNet: Removed LNI 192.168.201.136@tcp [ 8915.595269] Key type .llcrypt unregistered [ 8915.601297] Key type ._llcrypt unregistered [ 8934.123597] Key type ._llcrypt registered [ 8934.125870] Key type .llcrypt registered [ 8934.226330] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8935.175559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8935.232059] alg: No test for adler32 (adler32-zlib) [ 8936.420557] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 8936.662650] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8938.304423] Key type lgssc registered [ 8939.640056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8947.234912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8947.269434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8948.920196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8953.925611] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8959.458971] LustreError: 278466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8959.474957] LustreError: 278466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8962.030776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8962.405093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8967.350307] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8974.748971] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8980.374560] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8987.327469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8987.617259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8993.658291] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8993.795267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9002.388889] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9020.170656] Lustre: server umount lustre-OST0000 complete [ 9027.567858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9027.587630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9027.595379] Lustre: Skipped 1 previous similar message [ 9027.612912] Lustre: Skipped 1 previous similar message [ 9032.673606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9032.682562] Lustre: Skipped 1 previous similar message [ 9034.808132] Lustre: server umount lustre-MDT0000 complete [ 9037.791382] Lustre: 277763:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787588588/real 1787588588] req@0000000072462481 x1874422401102592/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787588604 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9037.799764] LustreError: 278479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9037.839581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9037.877626] LustreError: 278479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9038.982364] LustreError: 278432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588605 with bad export cookie 8203982920374409737 [ 9039.005047] LustreError: 278432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9039.006192] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9039.467741] Lustre: server umount lustre-MDT0001 complete [ 9051.358106] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9054.230421] Key type lgssc unregistered [ 9054.594737] LNet: 282169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9054.606644] LNetError: 282169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9054.628590] LNet: Removed LNI 192.168.201.136@tcp [ 9055.595721] Key type .llcrypt unregistered [ 9055.598951] Key type ._llcrypt unregistered [ 9068.002127] Key type ._llcrypt registered [ 9068.003914] Key type .llcrypt registered [ 9068.129192] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9086.318332] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 9103.145277] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9104.298470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9104.475240] alg: No test for adler32 (adler32-zlib) [ 9105.591889] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 9105.796401] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9107.463156] Key type lgssc registered [ 9108.702536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9147.006058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9147.055113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9148.525525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9148.605044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9148.795262] Lustre: lustre-MDT0000: new disk, initializing [ 9148.922297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9148.958228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9153.751499] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9165.388673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9165.576324] Lustre: 285724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9165.627092] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9165.630865] Lustre: Skipped 1 previous similar message [ 9165.685346] Lustre: lustre-MDT0001: new disk, initializing [ 9165.775234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9165.818051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9165.830233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9170.548933] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9180.206848] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9186.300366] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9193.647495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9193.883218] Lustre: lustre-OST0000: new disk, initializing [ 9193.891113] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9193.901097] Lustre: 287691:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9194.008549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9200.646769] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9202.737022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9202.761952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9202.886490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9211.207724] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9217.593090] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9217.922468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9222.950948] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9223.228046] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9228.781896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9228.784767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9228.794340] LustreError: Skipped 1 previous similar message [ 9228.794941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9228.809054] Lustre: Skipped 1 previous similar message [ 9231.526491] Lustre: server umount lustre-OST0000 complete [ 9237.477573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9237.486183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9237.497979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9237.502549] Lustre: Skipped 1 previous similar message [ 9239.009903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9239.010429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9239.039436] Lustre: Skipped 1 previous similar message [ 9241.323890] Lustre: server umount lustre-MDT0000 complete [ 9244.918708] LustreError: 285715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588811 with bad export cookie 12414122113470930852 [ 9244.930499] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9244.945848] LustreError: 285715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9245.370216] Lustre: server umount lustre-MDT0001 complete [ 9254.859868] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:26:59 (1787588819) [ 9261.325944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9261.684187] LustreError: 290103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9261.697215] LustreError: 290103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9261.747290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9266.009110] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9267.171962] LustreError: 290104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9272.289834] LustreError: 290103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9275.077744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9275.365502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9278.995970] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9285.980658] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9290.764722] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9296.700565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9296.974381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9302.341719] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9309.019338] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9317.861374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9317.863673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9317.864759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9317.872130] Lustre: Skipped 1 previous similar message [ 9323.030937] Lustre: server umount lustre-OST0000 complete [ 9326.560089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9326.566289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9326.575962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9326.578611] Lustre: Skipped 1 previous similar message [ 9332.835558] Lustre: server umount lustre-MDT0000 complete [ 9333.225849] LustreError: 290104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9333.234285] LustreError: 290104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9336.249050] LustreError: 290084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588902 with bad export cookie 12414122113470931734 [ 9336.255252] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9336.261965] LustreError: 290084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9336.620721] Lustre: server umount lustre-MDT0001 complete [ 9347.034322] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9349.361070] Key type lgssc unregistered [ 9349.633246] LNet: 293639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9349.638690] LNetError: 293639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9349.649296] LNet: Removed LNI 192.168.201.136@tcp [ 9350.447076] Key type .llcrypt unregistered [ 9350.448581] Key type ._llcrypt unregistered [ 9374.564877] Key type ._llcrypt registered [ 9374.566864] Key type .llcrypt registered [ 9374.694879] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 9388.230189] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9389.252767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9389.335882] alg: No test for adler32 (adler32-zlib) [ 9390.356738] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 9390.548399] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9392.178484] Key type lgssc registered [ 9393.440518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9436.702753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9436.742662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9438.004587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9438.039669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9438.107587] Lustre: lustre-MDT0000: new disk, initializing [ 9438.165098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9438.177979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9442.012254] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9452.178116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9452.352561] Lustre: 297461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9452.426590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9452.432558] Lustre: Skipped 1 previous similar message [ 9452.598320] Lustre: lustre-MDT0001: new disk, initializing [ 9452.780174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9452.860195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9452.879751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9457.734112] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9468.048691] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9474.874643] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9482.020426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9482.283427] Lustre: lustre-OST2710: new disk, initializing [ 9482.286077] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9482.291840] Lustre: 299431:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9482.363831] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9488.932316] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9492.535247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9492.542779] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9492.639515] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9499.693446] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9508.653507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9509.006215] Lustre: lustre-OST03e8: new disk, initializing [ 9509.021869] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9509.032140] Lustre: 300496:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9509.252357] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9515.153715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9515.177583] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9515.366356] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9517.730906] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9525.626785] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9533.728122] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9534.106979] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9540.300036] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9540.706513] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9546.277936] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9546.539356] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9551.845026] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9552.143629] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9559.071825] Lustre: server umount lustre-MDT0000 complete [ 9560.555994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9560.569571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9561.571133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9561.605801] Lustre: Skipped 2 previous similar messages [ 9563.727967] LustreError: 297454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589130 with bad export cookie 2122985026461646667 [ 9563.733417] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9564.548141] Lustre: server umount lustre-MDT0001 complete [ 9580.232879] Lustre: server umount lustre-OST2710 complete [ 9583.073213] Lustre: 295262:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787589133/real 1787589133] req@00000000a06bbded x1874422877198720/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787589149 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9583.096442] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9584.307510] Lustre: server umount lustre-OST03e8 complete [ 9604.833242] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 9616.349813] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9657.101207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9657.346622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9657.392103] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9657.482866] Lustre: lustre-MDT0000: new disk, initializing [ 9657.559186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9657.579526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9663.178930] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9675.695645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9675.826184] Lustre: 305748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9675.891547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9675.894881] Lustre: Skipped 1 previous similar message [ 9675.995286] Lustre: lustre-MDT0001: new disk, initializing [ 9676.130566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9676.162270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9676.192980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9682.488870] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9693.609610] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9700.608521] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9707.240594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9707.483423] Lustre: lustre-OST0000: new disk, initializing [ 9707.493360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9707.502963] Lustre: 307720:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9707.590533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9714.103499] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9715.824030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9715.832341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9716.001774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9725.076728] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9730.605141] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9730.837843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9735.692466] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9736.032498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9741.796417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9741.797371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9741.807141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9741.844565] Lustre: Skipped 1 previous similar message [ 9744.478392] Lustre: server umount lustre-OST0000 complete [ 9752.033868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9752.040087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9752.042435] Lustre: Skipped 2 previous similar messages [ 9754.835432] Lustre: server umount lustre-MDT0000 complete [ 9757.153606] LustreError: 306778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9757.180550] LustreError: 306778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9758.631937] LustreError: 305739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589324 with bad export cookie 2122985026461648347 [ 9758.640290] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9758.643086] LustreError: 305739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9758.957963] Lustre: server umount lustre-MDT0001 complete [ 9768.054723] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:35:33 (1787589333) [ 9769.554548] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9771.786503] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:35:36 (1787589336) [ 9775.958734] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9806.899280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9807.218950] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9807.243786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9807.435813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9812.824474] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9822.254701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9822.332552] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9822.361933] Lustre: 311575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9827.809140] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9851.326291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9851.576149] Lustre: Denying initial registration attempt from nid 192.168.201.136@tcp, specified as failover [ 9851.586279] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9851.599427] LustreError: 313031:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9851.604550] LustreError: 313031:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9851.625753] LustreError: 313031:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9851.737536] Lustre: server umount lustre-OST0000 complete [ 9851.742882] LustreError: 313031:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9859.044125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9859.065737] Lustre: Skipped 1 previous similar message [ 9859.070581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9859.084100] Lustre: Skipped 1 previous similar message [ 9864.161815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9864.171276] Lustre: Skipped 1 previous similar message [ 9869.281141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9869.289790] Lustre: Skipped 1 previous similar message [ 9873.376491] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9873.561813] Lustre: server umount lustre-MDT0000 complete [ 9874.401811] LustreError: 311583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9874.431469] LustreError: 311583:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9877.446313] LustreError: 311570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589443 with bad export cookie 2122985026461649229 [ 9877.456274] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9877.458052] LustreError: 311570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9877.870385] Lustre: server umount lustre-MDT0001 complete [ 9888.716895] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9891.943660] Key type lgssc unregistered [ 9892.270216] LNet: 314156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9892.274670] LNetError: 314156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9892.293472] LNet: Removed LNI 192.168.201.136@tcp [ 9893.207471] Key type .llcrypt unregistered [ 9893.210159] Key type ._llcrypt unregistered [ 9904.347906] Key type ._llcrypt registered [ 9904.354715] Key type .llcrypt registered [ 9904.459456] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9905.581768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9905.627108] alg: No test for adler32 (adler32-zlib) [ 9906.622835] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 9906.828390] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9908.559434] Key type lgssc registered [ 9909.611640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9960.557382] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9970.031310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9970.044124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9971.340749] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9971.364240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9971.576113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9976.157482] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9985.622985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9985.824793] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9985.878336] Lustre: 316922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9986.173981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9990.032357] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9997.237206] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10004.236931] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10025.261749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10025.472371] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10025.755716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10031.864994] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10039.933852] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10043.363387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10043.371513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10043.391426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10043.877479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10043.892544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10043.917578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10048.328837] Lustre: server umount lustre-OST0000 complete [10052.576316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10052.587523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10052.602986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10058.956089] Lustre: server umount lustre-MDT0000 complete [10063.003875] LustreError: 316914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589629 with bad export cookie 1500407511250093106 [10063.012304] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10063.015532] LustreError: 316914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10063.287726] Lustre: server umount lustre-MDT0001 complete [10074.624710] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [10077.045191] Key type lgssc unregistered [10077.412595] LNet: 320501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10077.431822] LNetError: 320501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10078.502165] LNet: Removed LNI 192.168.201.136@tcp [10079.306291] Key type .llcrypt unregistered [10079.309113] Key type ._llcrypt unregistered [10104.575861] Key type ._llcrypt registered [10104.581450] Key type .llcrypt registered [10104.686109] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10105.894819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10105.954171] alg: No test for adler32 (adler32-zlib) [10106.992348] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [10107.168083] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [10108.887296] Key type lgssc registered [10109.892796] Lustre: Echo OBD driver; http://www.lustre.org/ [10117.380424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10117.413059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10118.962594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10123.635659] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10129.376826] LustreError: 321920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.403499] LustreError: 321920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10132.188254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10132.628841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10137.488484] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10146.225162] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10154.458055] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10164.360360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10164.881208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10172.211936] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10181.572527] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10203.884972] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.36@tcp (stopping) [10204.128618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10204.135673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10205.160330] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10205.181941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10205.192359] Lustre: Skipped 3 previous similar messages [10206.951697] Lustre: server umount lustre-MDT0000 complete [10208.976232] LustreError: 321915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10210.279419] LustreError: 321914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10210.287093] LustreError: 321914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10210.771851] LustreError: 321899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589777 with bad export cookie 14843529126861265556 [10210.781862] LustreError: 321899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10210.782262] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10214.094419] LustreError: 321916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10214.125374] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [10215.397042] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10215.416226] Lustre: Skipped 1 previous similar message [10219.210703] LustreError: 324420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10219.214459] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [10219.226339] LustreError: 324420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10219.250306] Lustre: Skipped 1 previous similar message [10224.329378] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [10224.340261] Lustre: Skipped 2 previous similar messages [10225.121359] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10225.653178] Lustre: server umount lustre-MDT0001 complete [10232.312112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10241.829683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10256.644253] LustreError: 323751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.664111] LustreError: 323751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10256.757990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10256.815784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10261.451774] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10271.032571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10271.499498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10276.652305] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10277.071709] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10286.581762] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10288.623326] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10294.777826] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10297.159833] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10307.568251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10307.570724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10307.587415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10307.613920] Lustre: Skipped 1 previous similar message [10309.808392] Lustre: server umount lustre-OST0000 complete [10317.793028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10317.801906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10317.810474] Lustre: Skipped 1 previous similar message [10320.186178] Lustre: server umount lustre-MDT0000 complete [10322.915589] LustreError: 325536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10322.939828] LustreError: 325536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10324.750259] LustreError: 321899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589891 with bad export cookie 14843529126861273270 [10324.766946] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10324.771391] LustreError: 321899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10325.217041] Lustre: server umount lustre-MDT0001 complete [10341.537379] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [10344.698216] Key type lgssc unregistered [10344.965772] LNet: 328213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10344.973978] LNetError: 328213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10344.993159] LNet: Removed LNI 192.168.201.136@tcp [10345.920925] Key type .llcrypt unregistered [10345.922894] Key type ._llcrypt unregistered [10374.880876] Key type ._llcrypt registered [10374.883645] Key type .llcrypt registered [10374.961659] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10376.394145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10376.428768] alg: No test for adler32 (adler32-zlib) [10377.663248] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [10377.948176] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [10379.695216] Key type lgssc registered [10381.100098] Lustre: Echo OBD driver; http://www.lustre.org/ [10390.439589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10390.491664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10392.197949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10397.624706] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10401.776178] LustreError: 329633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.788644] LustreError: 329633:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10406.845413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10406.881404] LustreError: 329632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10407.337046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10412.858168] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10420.892517] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10427.380465] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10432.358529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10432.384619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10432.394959] Lustre: Skipped 1 previous similar message [10436.585628] Lustre: server umount lustre-MDT0000 complete [10437.603779] LustreError: 329633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10437.630855] LustreError: 329633:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10441.165033] LustreError: 329613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590007 with bad export cookie 8244871454279645281 [10441.178894] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10441.191366] LustreError: 329613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10441.703696] Lustre: server umount lustre-MDT0001 complete [10458.114317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10458.270842] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10458.294515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10458.458929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10463.143235] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10471.256551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10471.328410] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10471.391575] Lustre: 332222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10471.752811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10476.658378] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10484.434882] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10490.660862] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10501.307512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10501.571075] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10501.578476] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10501.585182] LustreError: 334107:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [10501.589744] LustreError: 334107:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10501.595246] LustreError: 334107:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10501.668606] Lustre: server umount lustre-OST0000 complete [10501.671532] LustreError: 334107:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10512.623688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10512.892956] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10513.143754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10518.260491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10519.992809] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10527.749860] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10535.540663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10535.645123] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10535.748798] Lustre: lustre-OST0001: new disk, initializing [10535.752230] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10535.759874] Lustre: 335647:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10535.878974] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10541.777056] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10543.217307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10543.239130] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10543.408148] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10549.411619] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10553.827685] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10553.828816] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.838111] LustreError: Skipped 1 previous similar message [10553.841988] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10553.851211] Lustre: Skipped 2 previous similar messages [10553.865592] Lustre: Skipped 1 previous similar message [10558.973984] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10558.988373] Lustre: Skipped 1 previous similar message [10564.074347] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10564.091996] Lustre: Skipped 1 previous similar message [10567.136059] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10567.329845] Lustre: server umount lustre-OST0001 complete [10569.199237] LustreError: 334576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10569.225800] LustreError: 334576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10574.310671] LustreError: 336107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10574.317676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10574.337482] LustreError: 336107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10574.347996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10578.329953] Lustre: server umount lustre-OST0000 complete [10584.544672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10584.545392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10584.560979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10584.560987] Lustre: Skipped 1 previous similar message [10584.612492] Lustre: Skipped 2 previous similar messages [10588.008995] Lustre: server umount lustre-MDT0000 complete [10589.667288] LustreError: 332229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10589.694992] LustreError: 332229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10591.934698] LustreError: 332215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590158 with bad export cookie 8244871454279645834 [10591.936634] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10591.942583] LustreError: 332215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10592.562988] Lustre: server umount lustre-MDT0001 complete [10605.330994] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [10608.101205] Key type lgssc unregistered [10608.336607] LNet: 337609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10608.344039] LNetError: 337609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10608.360553] LNet: Removed LNI 192.168.201.136@tcp [10609.339537] Key type .llcrypt unregistered [10609.341904] Key type ._llcrypt unregistered [10687.948895] Key type ._llcrypt registered [10687.953717] Key type .llcrypt registered [10688.126566] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [10705.458018] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10706.968241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10706.991434] alg: No test for adler32 (adler32-zlib) [10708.083976] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [10708.287482] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [10709.927166] Key type lgssc registered [10711.089711] Lustre: Echo OBD driver; http://www.lustre.org/ [10751.051346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10751.121817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10752.550444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10752.624820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10752.798927] Lustre: lustre-MDT0000: new disk, initializing [10752.923104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10752.949591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10759.540579] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10771.977744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10772.064927] Lustre: 342545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10772.118684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10772.124297] Lustre: Skipped 1 previous similar message [10772.207415] Lustre: lustre-MDT0001: new disk, initializing [10772.303203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10772.343102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10772.360855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10777.606863] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10788.504671] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10794.561640] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10801.867690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10802.110881] Lustre: lustre-OST0000: new disk, initializing [10802.116345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10802.124449] Lustre: 344517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10802.220765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10808.899904] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10810.004634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10810.017254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10810.155750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10818.562762] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10823.239485] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10823.464405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10828.716314] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10828.989851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10830.831881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10830.838369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10830.847378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10835.951773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10835.958212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10835.965422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10841.069581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10841.082134] Lustre: Skipped 2 previous similar messages [10845.151179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10845.430598] Lustre: server umount lustre-OST0000 complete [10851.300692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10851.317297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10851.326555] Lustre: Skipped 1 previous similar message [10851.355601] Lustre: Skipped 1 previous similar message [10855.656947] Lustre: server umount lustre-MDT0000 complete [10859.174486] LustreError: 342538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590425 with bad export cookie 10392182106903363207 [10859.186191] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10859.198635] LustreError: 342538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10859.635118] Lustre: server umount lustre-MDT0001 complete [10868.466447] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:53:53 (1787590433) [10876.499082] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:54:01 (1787590441) [10885.598797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10886.074028] LustreError: 347275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10886.093153] LustreError: 347275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10886.147689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10891.242467] LustreError: 347276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10891.603131] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10896.356159] LustreError: 347275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10901.131933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10901.508282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10907.061247] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10915.640688] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10922.449847] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10929.689726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10930.050460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10937.036365] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10944.896572] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10953.043393] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10954.994525] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10956.732333] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10958.571939] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10960.924404] Lustre: Failing over lustre-MDT0000 [10961.340571] Lustre: server umount lustre-MDT0000 complete [10961.897298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10961.903765] LustreError: 347272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10961.908182] Lustre: Skipped 1 previous similar message [10961.927734] LustreError: 347272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10967.015572] LustreError: 349117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10967.036251] LustreError: 349117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10977.259622] LustreError: 347270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10977.287426] LustreError: 347270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10978.271377] Lustre: 340771:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787590528/real 1787590528] req@00000000930c9179 x1874424259222528/t0(0) o400->MGC192.168.201.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787590544 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10978.309333] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10980.486718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10989.544672] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x90387290eb254613 [10989.561243] Lustre: MGC192.168.201.136@tcp: Connection restored to 0@lo (at 0@lo) [10989.858700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10989.929796] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10994.378228] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10995.000223] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10995.192111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10995.238845] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10995.299839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11005.483922] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11007.523455] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11011.447426] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11013.483720] Lustre: Failing over lustre-MDT0000 [11014.006057] Lustre: server umount lustre-MDT0000 complete [11014.928148] LustreError: 347270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11014.951717] LustreError: 347270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [11015.649534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11015.670787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11015.691311] Lustre: Skipped 2 previous similar messages [11032.031315] Lustre: 340772:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787590582/real 1787590582] req@00000000dde255be x1874424259244672/t0(0) o400->MGC192.168.201.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787590598 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11032.062867] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11032.634668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11042.290330] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x90387290eb254923 [11042.310953] Lustre: MGC192.168.201.136@tcp: Connection restored to 0@lo (at 0@lo) [11042.328254] Lustre: Skipped 2 previous similar messages [11042.522151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11042.600689] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11046.600090] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11047.568314] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11047.922376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11047.957094] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11047.996990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11056.557436] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11058.250762] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11068.391104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11068.396582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11068.402712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11068.406374] Lustre: Skipped 2 previous similar messages [11071.164455] Lustre: server umount lustre-MDT0000 complete [11073.507384] LustreError: 347270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11073.524356] LustreError: 347270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [11075.754156] LustreError: 351242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590642 with bad export cookie 10392182106903365923 [11075.757759] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11075.760066] LustreError: 351242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11076.033807] Lustre: server umount lustre-MDT0001 complete [11091.681467] Lustre: server umount lustre-OST0000 complete [11110.144439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11110.850471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11114.687400] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11124.983089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11131.096085] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11140.034646] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11146.355997] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11153.493333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11155.837461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11159.552856] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11166.598568] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11176.418277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11176.426373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11176.437399] Lustre: Skipped 1 previous similar message [11176.442351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11176.449963] Lustre: Skipped 2 previous similar messages [11177.960883] Lustre: server umount lustre-OST0000 complete [11188.125881] Lustre: server umount lustre-MDT0000 complete [11190.753681] LustreError: 353625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11190.772724] LustreError: 353625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11192.273950] LustreError: 353606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590758 with bad export cookie 10392182106903366658 [11192.282760] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11202.901850] Lustre: server umount lustre-MDT0001 complete [11215.698319] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11218.505902] Key type lgssc unregistered [11218.804089] LNet: 357159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11218.808603] LNetError: 357159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11218.840970] LNet: Removed LNI 192.168.201.136@tcp [11219.822897] Key type .llcrypt unregistered [11219.824750] Key type ._llcrypt unregistered [11250.160688] Key type ._llcrypt registered [11250.166744] Key type .llcrypt registered [11250.368088] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11251.848971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11251.875045] alg: No test for adler32 (adler32-zlib) [11252.975268] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [11253.256066] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11254.976627] Key type lgssc registered [11256.474345] Lustre: Echo OBD driver; http://www.lustre.org/ [11264.199462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11264.213236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11265.806229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11270.494901] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11276.260169] LustreError: 358679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11276.289988] LustreError: 358679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11279.115805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11279.468833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11283.966755] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11292.594418] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11299.767978] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11308.006694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11308.445157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11311.547668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11315.237771] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11324.272259] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11335.651573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11335.656712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11335.666814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11339.234599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11339.235564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11339.255948] Lustre: Skipped 1 previous similar message [11339.269628] Lustre: Skipped 1 previous similar message [11341.284097] Lustre: server umount lustre-MDT0000 complete [11344.356849] LustreError: 358674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11344.417102] LustreError: 358674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11346.696475] LustreError: 358660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590913 with bad export cookie 6126543640229165113 [11346.709464] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11346.716925] LustreError: 358660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11347.301367] Lustre: server umount lustre-MDT0001 complete [11363.349668] Lustre: server umount lustre-OST0000 complete [11378.283798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11378.997939] LustreError: 362405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11379.180567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11383.881781] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11384.290611] LustreError: 362406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11384.310402] LustreError: 362406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11389.413757] LustreError: 362405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11393.365456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11393.678230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11399.127825] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11407.473602] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11415.555993] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11423.903061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11424.525892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11425.648777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11431.796737] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11440.171829] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11453.418019] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11453.426710] Lustre: Skipped 1 previous similar message [11453.426891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11453.455321] Lustre: Skipped 3 previous similar messages [11458.541717] LustreError: 362405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11458.555860] LustreError: 362405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11458.617304] Lustre: server umount lustre-MDT0000 complete [11462.311814] LustreError: 362385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591028 with bad export cookie 6126543640229166226 [11462.326551] LustreError: 362385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11462.327844] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11462.626237] Lustre: server umount lustre-MDT0001 complete [11477.221241] Lustre: server umount lustre-OST0000 complete [11493.614635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11494.220861] LustreError: 366319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11494.274497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11498.297811] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11507.162580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11512.432615] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11519.767310] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11525.733252] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11533.505887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11534.192271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11534.201970] Lustre: Skipped 1 previous similar message [11539.340317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11542.852507] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11552.895798] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11559.905640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11559.911546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11559.931747] Lustre: Skipped 1 previous similar message [11559.938759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11562.977462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11564.594348] Lustre: server umount lustre-OST0000 complete [11569.121074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11569.128081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11569.149917] Lustre: Skipped 1 previous similar message [11569.158235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11569.162646] Lustre: Skipped 2 previous similar messages [11574.584742] Lustre: server umount lustre-MDT0000 complete [11578.347772] LustreError: 367046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11578.367617] LustreError: 367046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11578.511126] LustreError: 366301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591144 with bad export cookie 6126543640229167248 [11578.518506] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11578.525511] LustreError: 366301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11578.926819] Lustre: server umount lustre-MDT0001 complete [11592.572751] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11595.730255] Key type lgssc unregistered [11596.050499] LNet: 369855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11596.070081] LNetError: 369855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11596.092595] LNet: Removed LNI 192.168.201.136@tcp [11596.943388] Key type .llcrypt unregistered [11596.945746] Key type ._llcrypt unregistered [11626.025362] Key type ._llcrypt registered [11626.030678] Key type .llcrypt registered [11626.262915] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11628.176577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11628.210069] alg: No test for adler32 (adler32-zlib) [11629.276409] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [11629.444625] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11631.159153] Key type lgssc registered [11632.273767] Lustre: Echo OBD driver; http://www.lustre.org/ [11640.678527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11640.709892] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11640.722759] LustreError: 371298:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11640.749586] LustreError: 371298:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11640.759156] LustreError: 371298:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11640.765447] LustreError: 371298:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11640.779117] LustreError: 371298:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11649.613733] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11649.623188] LustreError: 371695:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11649.655144] LustreError: 371695:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11649.671500] LustreError: 371695:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11649.686545] LustreError: 371695:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11649.705348] LustreError: 371695:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11675.449827] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11679.772306] Key type lgssc unregistered [11680.162402] LNet: 372704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11680.167954] LNetError: 372704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11680.189113] LNet: Removed LNI 192.168.201.136@tcp [11680.819683] Key type .llcrypt unregistered [11680.822267] Key type ._llcrypt unregistered [11702.479530] Key type ._llcrypt registered [11702.483918] Key type .llcrypt registered [11702.648215] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [11719.620440] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11721.187572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11721.217368] alg: No test for adler32 (adler32-zlib) [11722.591733] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [11722.945520] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11724.751340] Key type lgssc registered [11726.214820] Lustre: Echo OBD driver; http://www.lustre.org/ [11763.724712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11763.789655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11765.175293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11765.246391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11765.429529] Lustre: lustre-MDT0000: new disk, initializing [11765.553298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11765.575929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11770.453105] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11783.474629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11783.580703] Lustre: 375862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11783.621721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11783.627920] Lustre: Skipped 1 previous similar message [11783.700608] Lustre: lustre-MDT0001: new disk, initializing [11783.788031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11783.867919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11783.881057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11789.859701] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11800.437776] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11807.684403] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11816.082768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11816.601738] Lustre: lustre-OST0000: new disk, initializing [11816.608757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11816.622245] Lustre: 377833:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11816.772638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11822.231071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11822.252046] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11822.423924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11824.824471] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11835.995896] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11841.052319] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11841.316691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11846.710860] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11846.894445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11853.285874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11853.290607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11853.310296] LustreError: Skipped 1 previous similar message [11853.318324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11853.343831] Lustre: Skipped 1 previous similar message [11855.314412] Lustre: server umount lustre-OST0000 complete [11860.449057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11860.463312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11860.480417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11860.495744] Lustre: Skipped 1 previous similar message [11863.524754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11863.536094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11863.565179] Lustre: Skipped 1 previous similar message [11868.644733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11868.654206] Lustre: Skipped 1 previous similar message [11872.737038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11872.745317] Lustre: Skipped 1 previous similar message [11874.271325] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11874.470022] Lustre: server umount lustre-MDT0000 complete [11878.538660] LustreError: 375854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591444 with bad export cookie 17140561077200184466 [11878.559894] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11878.560271] LustreError: 375854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11879.033797] Lustre: server umount lustre-MDT0001 complete [11889.122658] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11891.338533] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:10:55 (1787591455) [11898.589919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11899.203080] LustreError: 380294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11899.229363] LustreError: 380294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11899.320078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11904.489431] LustreError: 380295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11904.738120] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11909.615550] LustreError: 380294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11914.725357] LustreError: 380295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.291756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11915.969806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11922.383213] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11934.904274] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11941.941367] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11950.001074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11950.606477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11958.783505] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11967.922229] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11976.981731] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11977.179196] Lustre: lustre-OST0001: new disk, initializing [11977.182499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11977.187108] Lustre: 383133:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11977.284742] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11985.700416] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11987.149281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11987.172445] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11987.379387] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11999.660716] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12007.661357] Lustre: lustre-OST0001: Not available for connect from 192.168.201.36@tcp (stopping) [12007.912455] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12007.914331] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12007.940101] Lustre: Skipped 1 previous similar message [12007.964928] LustreError: Skipped 1 previous similar message [12011.963495] Lustre: server umount lustre-OST0001 complete [12012.760105] LustreError: 382120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12023.274677] LustreError: 383343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12023.313950] LustreError: 383343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12032.482089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12032.505893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12032.540130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12032.554253] Lustre: Skipped 2 previous similar messages [12043.750613] LustreError: 382120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12043.767639] LustreError: 382120:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [12045.280181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12045.554292] Lustre: server umount lustre-OST0000 complete [12053.985648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12053.992139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12054.005850] Lustre: Skipped 2 previous similar messages [12056.652332] Lustre: server umount lustre-MDT0000 complete [12061.026524] LustreError: 382140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591627 with bad export cookie 17140561077200185348 [12061.034312] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12061.046158] LustreError: 382140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12067.972217] Lustre: server umount lustre-MDT0001 complete [12083.310964] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [12086.467407] Key type lgssc unregistered [12086.818818] LNet: 385129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12086.851651] LNetError: 385129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12086.891537] LNet: Removed LNI 192.168.201.136@tcp [12087.862699] Key type .llcrypt unregistered [12087.865234] Key type ._llcrypt unregistered [12147.967329] Key type ._llcrypt registered [12147.968865] Key type .llcrypt registered [12148.072802] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [12150.223534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12150.260775] alg: No test for adler32 (adler32-zlib) [12151.498319] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [12151.799709] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [12153.487519] Key type lgssc registered [12154.713958] Lustre: Echo OBD driver; http://www.lustre.org/ [12164.593811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12164.625406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12165.968891] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12166.032431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12166.409143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12172.213810] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12181.637755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12181.772339] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12181.833918] Lustre: 387927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12182.100200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12186.868758] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12195.923663] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12202.354833] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12207.585475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12207.606340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12207.630472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12209.634174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12209.635531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12209.676344] Lustre: Skipped 1 previous similar message [12214.758671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12214.766066] Lustre: Skipped 1 previous similar message [12219.360074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12219.677310] Lustre: server umount lustre-MDT0000 complete [12223.802303] LustreError: 387922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591790 with bad export cookie 5406368755475655250 [12223.810995] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12223.813271] LustreError: 387922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12224.119454] Lustre: server umount lustre-MDT0001 complete [12234.000705] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:16:38 (1787591798) [12241.049437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12241.431046] LustreError: 390442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12241.439961] LustreError: 390442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12241.486358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12245.558923] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12246.503833] LustreError: 390443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12251.623248] LustreError: 390442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12254.862462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12255.124865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12259.639935] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12268.133493] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12273.746835] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12281.684432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12281.917807] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12282.164837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12289.449425] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12297.606813] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12307.042649] Lustre: Permanently deactivating lustre-OST0000 [12307.069867] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12315.193298] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12316.965603] LustreError: 393431:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12316.971914] LustreError: 393431:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12316.976622] LustreError: 393431:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12321.249340] LustreError: 393480:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12321.252865] LustreError: 393480:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12329.388688] Lustre: server umount lustre-OST0000 complete [12333.115648] LustreError: 393730:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12333.127099] LustreError: 393730:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12336.097670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12336.104362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12336.115660] Lustre: Skipped 1 previous similar message [12336.152863] Lustre: Skipped 1 previous similar message [12341.220156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12341.235383] Lustre: Skipped 1 previous similar message [12345.431855] Lustre: server umount lustre-MDT0000 complete [12346.340548] LustreError: 390443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12346.364705] LustreError: 390443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12349.727928] LustreError: 390424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591916 with bad export cookie 5406368755475655831 [12349.729849] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12349.748420] LustreError: 390424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12360.397445] Lustre: server umount lustre-MDT0001 complete [12369.979939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12375.685978] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12379.168959] LustreError: 394890:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12385.636722] Lustre: 395230:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.136@tcp [12387.962212] Lustre: server umount MGS complete [12401.444820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12401.882561] LustreError: 395956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12401.907634] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12401.917442] Lustre: Skipped 1 previous similar message [12401.978675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12409.227413] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12412.412595] LustreError: 395957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12412.464370] LustreError: 395957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12420.650303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12420.866950] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12420.968345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12426.355659] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12435.705741] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12441.835094] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12443.812921] Lustre: Permanently reactivating lustre-OST0000 [12443.825391] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12443.829686] Lustre: Skipped 2 previous similar messages [12469.730683] LustreError: 397689:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12478.428935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12478.774634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12485.289284] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12493.821910] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12500.497659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12500.510946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12500.534215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12500.546453] Lustre: Skipped 1 previous similar message [12500.567559] Lustre: Skipped 1 previous similar message [12506.134657] Lustre: server umount lustre-OST0000 complete [12513.251322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12513.268964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12518.647635] Lustre: server umount lustre-MDT0000 complete [12520.932167] LustreError: 395952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12520.960919] LustreError: 395952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12522.739579] LustreError: 395936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592089 with bad export cookie 5406368755475656881 [12522.753506] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12523.394783] Lustre: server umount lustre-MDT0001 complete [12537.797573] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [12540.604797] Key type lgssc unregistered [12540.937512] LNet: 400276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12540.942223] LNetError: 400276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12540.965952] LNet: Removed LNI 192.168.201.136@tcp [12541.811793] Key type .llcrypt unregistered [12541.813870] Key type ._llcrypt unregistered [12562.186512] Key type ._llcrypt registered [12562.189119] Key type .llcrypt registered [12562.278826] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [12579.016371] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [12580.535820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12580.562956] alg: No test for adler32 (adler32-zlib) [12581.531956] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [12581.735676] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [12583.383456] Key type lgssc registered [12584.700409] Lustre: Echo OBD driver; http://www.lustre.org/ [12629.656848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12629.677574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12630.952428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12630.996984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12631.115838] Lustre: lustre-MDT0000: new disk, initializing [12631.207429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12631.226230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12636.305591] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12647.751252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12647.867538] Lustre: 403436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12647.907526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12647.911931] Lustre: Skipped 1 previous similar message [12648.009217] Lustre: lustre-MDT0001: new disk, initializing [12648.114952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12648.169665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12648.179109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12652.901830] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12664.066390] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12670.593961] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12677.378233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12677.740852] Lustre: lustre-OST0000: new disk, initializing [12677.750614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12677.759269] Lustre: 405404:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12677.862128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12685.782548] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12685.977166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12685.993497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12686.146433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12696.664721] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12702.229738] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12702.676198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12708.652236] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12708.945981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12711.904443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12711.907749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12711.925454] Lustre: Skipped 1 previous similar message [12711.947602] Lustre: Skipped 1 previous similar message [12717.026073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12717.045824] Lustre: Skipped 1 previous similar message [12722.172935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12726.246450] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12726.550480] Lustre: server umount lustre-OST0000 complete [12730.336854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12730.347667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12730.359754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12730.370480] Lustre: Skipped 1 previous similar message [12732.391836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12736.407816] Lustre: server umount lustre-MDT0000 complete [12740.879993] LustreError: 403427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592307 with bad export cookie 3215614079823834462 [12740.889954] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12740.894074] LustreError: 403427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12741.209449] Lustre: server umount lustre-MDT0001 complete [12753.215299] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12755.634267] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:25:19 (1787592319) [12766.030253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12766.844845] LustreError: 407867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12766.867435] LustreError: 407867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12767.030915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12772.346990] LustreError: 407868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12772.365435] LustreError: 407868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12773.861328] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12777.450766] LustreError: 407867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12783.591639] LustreError: 407867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12783.627136] LustreError: 407867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12785.666662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12786.034694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12791.970760] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12802.330350] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12810.311156] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12819.290574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12819.728022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12829.107783] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12838.842524] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12840.656702] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12843.196812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12848.609357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12848.614441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12848.623740] Lustre: Skipped 1 previous similar message [12848.628581] Lustre: Skipped 3 previous similar messages [12850.060814] Lustre: server umount lustre-OST0000 complete [12857.838871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12857.843041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12857.853837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12868.063168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12868.235399] Lustre: server umount lustre-MDT0000 complete [12869.089735] LustreError: 410084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12869.121636] LustreError: 410084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12872.331413] LustreError: 407849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592438 with bad export cookie 3215614079823835344 [12872.340709] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12872.342307] LustreError: 407849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12872.615684] Lustre: server umount lustre-MDT0001 complete [12886.037222] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [12889.233480] Key type lgssc unregistered [12889.597664] LNet: 411598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12889.602146] LNetError: 411598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12889.619850] LNet: Removed LNI 192.168.201.136@tcp [12890.680106] Key type .llcrypt unregistered [12890.702105] Key type ._llcrypt unregistered [12923.026698] Key type ._llcrypt registered [12923.029843] Key type .llcrypt registered [12923.176315] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [12923.557574] Key type .llcrypt unregistered [12923.560698] Key type ._llcrypt unregistered [12940.528414] Key type ._llcrypt registered [12940.532112] Key type .llcrypt registered [12940.653336] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [12941.713602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12941.895413] alg: No test for adler32 (adler32-zlib) [12943.062768] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [12943.276163] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [12944.951367] Key type lgssc registered [12946.181053] Lustre: Echo OBD driver; http://www.lustre.org/ [12953.390958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12953.404676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12954.890608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12960.104674] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12965.364406] LustreError: 413827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12965.376564] LustreError: 413827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12968.543506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12968.910906] LustreError: 413828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12968.925813] LustreError: 413828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12968.932414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12974.057104] LustreError: 413827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12974.070849] LustreError: 413827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12975.319424] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12979.179316] LustreError: 413828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12979.198254] LustreError: 413828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12980.245175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12983.560340] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12985.314036] LustreError: 413823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12985.327159] LustreError: 413823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12990.269461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12990.434648] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12990.439614] Lustre: Skipped 1 previous similar message [12990.514337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12994.438390] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13005.794357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13005.797829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13005.817709] Lustre: Skipped 1 previous similar message [13010.917408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13010.940867] Lustre: Skipped 2 previous similar messages [13011.473757] Lustre: server umount lustre-OST0000 complete [13021.153271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13021.168420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13021.178196] Lustre: Skipped 1 previous similar message [13021.193474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13024.007753] Lustre: server umount lustre-MDT0000 complete [13026.277164] LustreError: 413827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13026.309993] LustreError: 413827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13028.206058] LustreError: 413809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592594 with bad export cookie 10130300063159200752 [13028.217658] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13028.225164] LustreError: 413809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13028.714891] Lustre: server umount lustre-MDT0001 complete [13041.353240] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13045.096902] Key type lgssc unregistered [13045.533189] LNet: 416978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13045.543521] LNetError: 416978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13045.554738] LNet: Removed LNI 192.168.201.136@tcp [13046.790263] Key type .llcrypt unregistered [13046.793041] Key type ._llcrypt unregistered [13073.032544] Key type ._llcrypt registered [13073.034695] Key type .llcrypt registered [13073.129691] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [13074.036613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13074.053330] alg: No test for adler32 (adler32-zlib) [13075.082270] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [13075.295735] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [13076.943211] Key type lgssc registered [13077.838327] Lustre: Echo OBD driver; http://www.lustre.org/ [13085.630560] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13085.654695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13101.215562] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13106.335654] LustreError: 418367:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.136@tcp: failed processing log, type 4: rc = -110 [13133.024176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13139.828209] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13148.213550] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13156.840865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13157.289844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13157.344075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13162.289970] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13162.479630] LustreError: 418392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13162.504566] LustreError: 418392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13167.598993] LustreError: 418393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13168.627536] LustreError: 418392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13168.678537] LustreError: 418392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13172.508266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13173.133327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13179.536802] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13190.104052] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13198.774041] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13208.630993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13208.642628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13208.670330] LustreError: 418384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13208.840676] Lustre: server umount lustre-OST0000 complete [13214.189755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13214.212731] Lustre: Skipped 1 previous similar message [13214.213608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13214.213736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13215.366641] Lustre: server umount lustre-MDT0000 complete [13219.296805] LustreError: 419410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13219.316882] LustreError: 419410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13219.634949] LustreError: 418373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592785 with bad export cookie 2473389729771908883 [13219.640730] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13219.646180] LustreError: 418373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13229.146733] Lustre: server umount lustre-MDT0001 complete [13240.252394] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13243.539688] Key type lgssc unregistered [13243.880657] LNet: 421942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13243.888365] LNetError: 421942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13243.911548] LNet: Removed LNI 192.168.201.136@tcp [13244.882956] Key type .llcrypt unregistered [13244.884826] Key type ._llcrypt unregistered [13275.191430] Key type ._llcrypt registered [13275.192911] Key type .llcrypt registered [13275.354895] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [13276.773908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13277.178343] alg: No test for adler32 (adler32-zlib) [13278.478446] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [13278.976915] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [13280.745613] Key type lgssc registered [13282.161454] Lustre: Echo OBD driver; http://www.lustre.org/ [13291.426919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13291.457387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13293.116086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13298.458899] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13303.265322] LustreError: 423368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13303.303654] LustreError: 423368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13308.392582] LustreError: 423369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13308.520355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13308.892759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13313.443962] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13322.242651] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13329.463308] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13337.558642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13337.897882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13338.953619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13347.122438] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13358.527825] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13364.707252] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13364.712286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13364.720735] Lustre: Skipped 2 previous similar messages [13364.727898] Lustre: Skipped 2 previous similar messages [13367.015230] LustreError: 423365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13368.835179] Lustre: server umount lustre-MDT0000 complete [13369.829904] LustreError: 424372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13369.872135] LustreError: 424372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13371.347088] LustreError: 424102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.36@tcp arrived at 1787592937 with bad export cookie 16971722387318249881 [13371.367500] LustreError: 424102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13373.768503] LustreError: 423347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592940 with bad export cookie 16971722387318248985 [13373.785914] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13374.944101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13374.948543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13374.949212] LustreError: 425049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13374.959794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13374.983233] LustreError: 425049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13378.977044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13380.227895] Lustre: server umount lustre-OST0000 complete [13384.097401] LustreError: 425049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13384.134305] LustreError: 425049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13388.519759] Lustre: server umount lustre-MDT0001 complete [13400.658672] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13403.118687] Key type lgssc unregistered [13403.389420] LNet: 427058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13403.397513] LNetError: 427058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13403.417166] LNet: Removed LNI 192.168.201.136@tcp [13404.494106] Key type .llcrypt unregistered [13404.496864] Key type ._llcrypt unregistered [13433.944324] Key type ._llcrypt registered [13433.947759] Key type .llcrypt registered [13434.101748] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [13435.790101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13435.880345] alg: No test for adler32 (adler32-zlib) [13437.090280] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [13437.360032] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [13439.024261] Key type lgssc registered [13440.261759] Lustre: Echo OBD driver; http://www.lustre.org/ [13449.113766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13449.131529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13450.504913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13455.582709] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13459.943494] LustreError: 428481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13459.973648] LustreError: 428481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13465.067555] LustreError: 428480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13465.670170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13466.101727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13470.432635] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13479.735167] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13489.398526] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13499.236384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13499.999557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13508.002798] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13518.280518] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13520.511552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:193) [13530.351710] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [13531.146497] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13531.176945] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13531.191439] Lustre: Skipped 1 previous similar message [13534.922056] Lustre: server umount lustre-MDT0001 complete [13535.446526] LustreError: 428476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13541.356863] LustreError: 428480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13541.386841] LustreError: 428480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13546.465831] LustreError: 429208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13546.474297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13546.494496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13546.513433] Lustre: Skipped 1 previous similar message [13546.519529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13548.995184] Lustre: server umount lustre-OST0000 complete [13555.354375] Lustre: server umount lustre-MDT0000 complete [13574.396754] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13577.712904] Key type lgssc unregistered [13578.125982] LNet: 432175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13578.135221] LNetError: 432175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13578.147858] LNet: Removed LNI 192.168.201.136@tcp [13579.175275] Key type .llcrypt unregistered [13579.180544] Key type ._llcrypt unregistered [13611.491950] Key type ._llcrypt registered [13611.493205] Key type .llcrypt registered [13611.642157] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [13629.516064] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [13631.398739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13631.477910] alg: No test for adler32 (adler32-zlib) [13632.564182] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [13632.921355] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [13634.695182] Key type lgssc registered [13635.787054] Lustre: Echo OBD driver; http://www.lustre.org/ [13678.295781] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [13694.971369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13695.007808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13696.333923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13696.386687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13696.469874] Lustre: lustre-MDT0000: new disk, initializing [13696.600772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13696.661920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13701.726363] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13717.944680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13718.386767] Lustre: lustre-OST0000: new disk, initializing [13718.399651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13718.412995] Lustre: Skipped 1 previous similar message [13718.422345] Lustre: 437214:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13718.550852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13724.277048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13724.284893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13724.341430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13726.660229] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13738.806631] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13758.315099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13758.471759] Lustre: 436194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13758.544807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13758.616619] Lustre: lustre-MDT0001: new disk, initializing [13758.765126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13758.804602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13758.825269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13764.782804] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13777.026554] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13777.287441] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13782.790782] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13783.011788] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13791.201752] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13791.215503] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13791.241458] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13791.249423] Lustre: Skipped 1 previous similar message [13791.258975] Lustre: Skipped 1 previous similar message [13793.000502] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [13795.233664] Lustre: server umount lustre-MDT0001 complete [13801.442893] LustreError: 436207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13801.468304] LustreError: 436207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13806.567561] LustreError: 436208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13806.579482] LustreError: 436208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13811.682583] LustreError: 436203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13811.699611] LustreError: 436203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13816.293913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13816.297901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13816.305967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13816.802056] LustreError: 436207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13821.921720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13821.922531] LustreError: 436208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13822.275574] Lustre: server umount lustre-OST0000 complete [13840.351819] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13840.977265] Lustre: server umount lustre-MDT0000 complete [13858.383911] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13862.894952] Key type lgssc unregistered [13863.381864] LNet: 440964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13863.390495] LNetError: 440964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13863.408921] LNet: Removed LNI 192.168.201.136@tcp [13865.196513] Key type .llcrypt unregistered [13865.199294] Key type ._llcrypt unregistered [13923.097717] Key type ._llcrypt registered [13923.103224] Key type .llcrypt registered [13923.244324] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13923.687332] Key type .llcrypt unregistered [13923.691704] Key type ._llcrypt unregistered [13942.287390] Key type ._llcrypt registered [13942.290830] Key type .llcrypt registered [13942.419495] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [13943.516430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13943.691823] alg: No test for adler32 (adler32-zlib) [13944.768831] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [13945.007751] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [13946.711236] Key type lgssc registered [13947.944888] Lustre: Echo OBD driver; http://www.lustre.org/ [13979.047313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13979.095569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13980.553061] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13980.606894] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13980.785456] Lustre: lustre-MDT0000: new disk, initializing [13980.857387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13980.869894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13986.713612] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13999.771454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13999.939201] Lustre: 444354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14000.011263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14000.014788] Lustre: Skipped 1 previous similar message [14000.073559] Lustre: lustre-MDT0001: new disk, initializing [14000.166764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14000.217553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14000.224418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14006.385634] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14018.297889] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14025.473833] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14033.940827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14034.261829] Lustre: lustre-OST0000: new disk, initializing [14034.266906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14034.276464] Lustre: 446322:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14034.392948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14043.154477] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14044.347454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14044.378809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14044.651583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14056.570921] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14065.143377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14065.149506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14065.168219] Lustre: Skipped 1 previous similar message [14065.196661] Lustre: Skipped 2 previous similar messages [14070.247975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14070.270697] Lustre: Skipped 2 previous similar messages [14070.719954] Lustre: server umount lustre-MDT0000 complete [14075.365440] LustreError: 444364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14075.406192] LustreError: 444364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14075.734363] LustreError: 444345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787593642 with bad export cookie 1912461935943746704 [14075.752902] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14075.758595] LustreError: 444345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14076.375238] Lustre: server umount lustre-MDT0001 complete [14092.488372] Lustre: server umount lustre-OST0000 complete [14103.351961] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:47:48 (1787593668) [14111.718923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14112.346235] LustreError: 448353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14112.513616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14117.857605] LustreError: 448354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14118.571386] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14121.962143] LustreError: 448354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14127.074342] LustreError: 448354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14129.684288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14130.034970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14135.930756] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14146.598410] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14152.234899] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14176.575953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14176.801626] Lustre: Found index 0 for lustre-OST0000, updating log [14176.958392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14182.087522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14185.216855] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14195.029289] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14201.827516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14201.849348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14201.867639] Lustre: Skipped 1 previous similar message [14203.024209] LustreError: 450180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14203.373433] Lustre: server umount lustre-OST0000 complete [14222.306074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14222.318066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14222.327959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14222.330791] Lustre: Skipped 1 previous similar message [14224.425170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14227.543627] Lustre: server umount lustre-MDT0000 complete [14231.012267] LustreError: 448353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14231.038454] LustreError: 448353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14231.998913] LustreError: 448333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787593798 with bad export cookie 1912461935943748097 [14232.007891] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14232.020509] LustreError: 448333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [14232.545471] Lustre: server umount lustre-MDT0001 complete [14233.823630] LNet: 1 local NIs in recovery (showing 1): 192.168.201.136@tcp [14242.907666] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:50:07 (1787593807) [14875.096280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14875.750088] LustreError: 454764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14875.864958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14881.262809] LustreError: 454765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14881.760947] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14892.515874] LustreError: 454764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14892.559995] LustreError: 454764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14892.793175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14893.428283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14900.759339] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14910.583052] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14917.780820] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14925.025711] LNet: 1 local NIs in recovery (showing 1): 192.168.201.136@tcp [14940.410336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14940.961884] Lustre: Found index 0 for lustre-OST0000, updating log [14941.796305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14949.469146] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14960.713703] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14964.457778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14964.482540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14964.490864] Lustre: Skipped 4 previous similar messages [14969.741607] Lustre: server umount lustre-OST0000 complete [14975.332903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14975.352270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14975.359026] Lustre: Skipped 1 previous similar message [14976.416705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14976.432399] Lustre: Skipped 1 previous similar message [14978.467507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14978.473141] Lustre: Skipped 3 previous similar messages [14980.676810] Lustre: server umount lustre-MDT0000 complete [14981.543878] LustreError: 455697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14981.560519] LustreError: 455697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14984.937790] LustreError: 456633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787594551 with bad export cookie 1912461935943749168 [14984.941826] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14984.956199] LustreError: 456633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14985.465262] Lustre: server umount lustre-MDT0001 complete [14990.623158] LNet: 1 local NIs in recovery (showing 1): 192.168.201.136@tcp [15019.106711] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [15032.188457] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [15079.626057] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:04:04 (1787594644) [15099.946340] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [15103.044776] Key type lgssc unregistered [15103.338712] LNet: 461404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15103.354208] LNetError: 461404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15201.704414] LNet: Removed LNI 192.168.201.136@tcp [15202.929293] Key type .llcrypt unregistered [15202.930883] Key type ._llcrypt unregistered [15224.093673] Key type ._llcrypt registered [15224.098550] Key type .llcrypt registered [15224.246922] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [15225.873380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15225.920071] alg: No test for adler32 (adler32-zlib) [15227.382733] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [15227.700481] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [15229.519743] Key type lgssc registered [15231.120273] Lustre: Echo OBD driver; http://www.lustre.org/ [15236.930858] LNet: Added LNI 192.168.201.136@tcp100 [8/256/0/180] [15238.150774] LNet: Added LNI 192.168.201.136@tcp101 [8/256/0/180] [15239.340782] LNet: Added LNI 192.168.201.136@tcp102 [8/256/0/180] [15240.707512] LNet: Added LNI 192.168.201.136@tcp103 [8/256/0/180] [15242.040622] LNet: Added LNI 192.168.201.136@tcp104 [8/256/0/180] [15243.234316] LNet: Added LNI 192.168.201.136@tcp105 [8/256/0/180] [15244.269586] LNet: Added LNI 192.168.201.136@tcp106 [8/256/0/180] [15245.600963] LNet: Added LNI 192.168.201.136@tcp107 [8/256/0/180] [15246.720485] LNet: Added LNI 192.168.201.136@tcp108 [8/256/0/180] [15248.068721] LNet: Added LNI 192.168.201.136@tcp109 [8/256/0/180] [15249.008694] LNet: Added LNI 192.168.201.136@tcp110 [8/256/0/180] [15250.001593] LNet: Added LNI 192.168.201.136@tcp111 [8/256/0/180] [15251.219485] LNet: Added LNI 192.168.201.136@tcp112 [8/256/0/180] [15252.573141] LNet: Added LNI 192.168.201.136@tcp113 [8/256/0/180] [15253.630952] LNet: Added LNI 192.168.201.136@tcp114 [8/256/0/180] [15254.770533] LNet: Added LNI 192.168.201.136@tcp115 [8/256/0/180] [15256.174607] LNet: Added LNI 192.168.201.136@tcp116 [8/256/0/180] [15257.369296] LNet: Added LNI 192.168.201.136@tcp117 [8/256/0/180] [15258.638589] LNet: Added LNI 192.168.201.136@tcp118 [8/256/0/180] [15259.767330] LNet: Added LNI 192.168.201.136@tcp119 [8/256/0/180] [15261.129801] LNet: Added LNI 192.168.201.136@tcp120 [8/256/0/180] [15262.345258] LNet: Added LNI 192.168.201.136@tcp121 [8/256/0/180] [15263.376933] LNet: Added LNI 192.168.201.136@tcp122 [8/256/0/180] [15264.366777] LNet: Added LNI 192.168.201.136@tcp123 [8/256/0/180] [15265.664503] LNet: Added LNI 192.168.201.136@tcp124 [8/256/0/180] [15267.036620] LNet: Added LNI 192.168.201.136@tcp125 [8/256/0/180] [15268.262746] LNet: Added LNI 192.168.201.136@tcp126 [8/256/0/180] [15269.506288] LNet: Added LNI 192.168.201.136@tcp127 [8/256/0/180] [15270.631848] LNet: Added LNI 192.168.201.136@tcp128 [8/256/0/180] [15271.613751] LNet: Added LNI 192.168.201.136@tcp129 [8/256/0/180] [15272.853375] LNet: Added LNI 192.168.201.136@tcp130 [8/256/0/180] [15274.076391] LNet: Added LNI 192.168.201.136@tcp131 [8/256/0/180] [15275.040545] LNet: Added LNI 192.168.201.136@tcp132 [8/256/0/180] [15276.685937] LNet: Added LNI 192.168.201.136@tcp133 [8/256/0/180] [15278.105823] LNet: Added LNI 192.168.201.136@tcp134 [8/256/0/180] [15279.627045] LNet: Added LNI 192.168.201.136@tcp135 [8/256/0/180] [15287.573251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15287.602421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15288.713381] Lustre: 464441:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [15288.766890] LNetError: 464462:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [15288.782062] LNetError: Protocol error connecting to 192.168.201.136@tcp123 at host 192.168.201.136:988: is it running a compatible version of Lustre? [15288.802170] LNetError: Skipped 6 previous similar messages [15289.012571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15289.048899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15289.134592] Lustre: lustre-MDT0000: new disk, initializing [15289.249808] LNet: 3 local NIs in recovery (showing 3): 192.168.201.136@tcp128, 192.168.201.136@tcp129, 192.168.201.136@tcp123 [15289.278191] LNetError: 464465:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [15289.291241] LNetError: 464465:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [15289.305663] LNetError: Protocol error connecting to 192.168.201.136@tcp102 at host 192.168.201.136:988: is it running a compatible version of Lustre? [15289.320840] LNetError: Skipped 8 previous similar messages [15289.339253] Lustre: 464500:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [15289.360242] Lustre: 464500:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15289.396861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15289.445245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15291.301355] LNetError: 464464:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [15291.324966] LNetError: 464464:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [15291.341669] LNetError: Protocol error connecting to 192.168.201.136@tcp128 at host 192.168.201.136:988: is it running a compatible version of Lustre? [15295.392311] LNetError: 464465:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [15295.401512] LNetError: Protocol error connecting to 192.168.201.136@tcp128 at host 192.168.201.136:988: is it running a compatible version of Lustre? [15295.567859] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15303.595716] LNetError: 461895:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [15303.627670] LNetError: Protocol error connecting to 192.168.201.136@tcp128 at host 192.168.201.136:988: is it running a compatible version of Lustre? [15310.393424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15310.561507] Lustre: 464458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15310.679509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15310.682787] Lustre: Skipped 1 previous similar message [15310.782577] Lustre: lustre-MDT0001: new disk, initializing [15310.866789] Lustre: 465265:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [15310.877438] Lustre: 465265:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15311.009547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15311.057682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15311.069696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15316.804129] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15317.599894] Lustre: 465547:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [15317.622388] Lustre: 465547:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15319.842917] LNetError: 464461:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [15319.857314] LNetError: Protocol error connecting to 192.168.201.136@tcp121 at host 192.168.201.136:988: is it running a compatible version of Lustre? [15328.558848] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15334.887062] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15336.228107] LNetError: 461896:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [15336.237268] LNetError: 461896:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 15 previous similar messages [15336.253580] LNetError: Protocol error connecting to 192.168.201.136@tcp122 at host 192.168.201.136:988: is it running a compatible version of Lustre? [15336.263091] LNetError: Skipped 15 previous similar messages [15343.101287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15343.533295] Lustre: lustre-OST0000: new disk, initializing [15343.538889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15343.545513] Lustre: 466438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15343.664495] Lustre: 466443:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [15343.676286] Lustre: 466443:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15343.747395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15349.663648] LNet: 1 local NIs in recovery (showing 1): 192.168.201.136@tcp128 [15351.197614] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15353.380318] Lustre: 466894:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [15353.391714] Lustre: 466894:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15353.457140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15353.470118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15353.609099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15362.744698] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15370.018600] LNetError: 464463:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [15370.025221] LNetError: 464463:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 25 previous similar messages [15370.037600] LNetError: Protocol error connecting to 192.168.201.136@tcp126 at host 192.168.201.136:988: is it running a compatible version of Lustre? [15370.059744] LNetError: Skipped 25 previous similar messages [15374.321743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15374.330507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15374.335981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15374.358668] Lustre: Skipped 1 previous similar message [15374.385845] LustreError: Skipped 1 previous similar message [15377.697193] Lustre: server umount lustre-OST0000 complete [15384.545334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15384.559093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15384.568255] Lustre: Skipped 1 previous similar message [15384.584369] Lustre: Skipped 2 previous similar messages [15389.674034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15389.698755] Lustre: Skipped 1 previous similar message [15393.830081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15393.832962] Lustre: Skipped 1 previous similar message [15397.343601] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15397.599947] Lustre: server umount lustre-MDT0000 complete [15402.616981] LustreError: 464449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787594968 with bad export cookie 5591785744372477080 [15402.637864] LustreError: 464449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15402.641355] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15402.948329] Lustre: server umount lustre-MDT0001 complete [15417.960412] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [15422.010231] Key type lgssc unregistered [15422.548804] LNet: 468356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15422.556035] LNetError: 468356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15422.562746] LNet: Removed LNI 192.168.201.136@tcp [15423.583969] LNet: Removed LNI 192.168.201.136@tcp100 [15423.586061] LNetError: 468356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15423.603063] LNetError: 468356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15423.617614] LNet: Removed LNI 192.168.201.136@tcp101 [15424.671210] LNet: Removed LNI 192.168.201.136@tcp102 [15424.677295] LNetError: 468356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15424.703534] LNetError: 468356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15424.718358] LNet: Removed LNI 192.168.201.136@tcp103 [15425.759207] LNet: Removed LNI 192.168.201.136@tcp104 [15426.783491] LNet: Removed LNI 192.168.201.136@tcp105 [15426.795419] LNetError: 468356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15426.816973] LNetError: 468356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15426.838949] LNet: Removed LNI 192.168.201.136@tcp106 [15427.871295] LNet: Removed LNI 192.168.201.136@tcp107 [15428.895690] LNet: Removed LNI 192.168.201.136@tcp108 [15429.921904] LNet: Removed LNI 192.168.201.136@tcp109 [15430.943209] LNet: Removed LNI 192.168.201.136@tcp110 [15430.970478] LNetError: 468356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15430.984026] LNetError: 468356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15431.001274] LNet: Removed LNI 192.168.201.136@tcp111 [15432.032679] LNet: Removed LNI 192.168.201.136@tcp112 [15433.055192] LNet: Removed LNI 192.168.201.136@tcp113 [15434.079740] LNet: Removed LNI 192.168.201.136@tcp114 [15435.104722] LNet: Removed LNI 192.168.201.136@tcp115 [15436.191151] LNet: Removed LNI 192.168.201.136@tcp116 [15437.215492] LNet: Removed LNI 192.168.201.136@tcp117 [15438.239256] LNet: Removed LNI 192.168.201.136@tcp118 [15439.264027] LNet: Removed LNI 192.168.201.136@tcp119 [15439.268634] LNetError: 468356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15439.290324] LNetError: 468356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15439.301835] LNet: Removed LNI 192.168.201.136@tcp120 [15440.351267] LNet: Removed LNI 192.168.201.136@tcp121 [15441.376039] LNet: Removed LNI 192.168.201.136@tcp122 [15442.401644] LNet: Removed LNI 192.168.201.136@tcp123 [15443.423392] LNet: Removed LNI 192.168.201.136@tcp124 [15444.447749] LNet: Removed LNI 192.168.201.136@tcp125 [15445.471333] LNet: Removed LNI 192.168.201.136@tcp126 [15446.495300] LNet: Removed LNI 192.168.201.136@tcp127 [15446.496978] LNet: Removed LNI 192.168.201.136@tcp128 [15447.519712] LNet: Removed LNI 192.168.201.136@tcp129 [15447.529320] LNet: Removed LNI 192.168.201.136@tcp130 [15447.548564] LNet: Removed LNI 192.168.201.136@tcp131 [15447.553733] LNet: Removed LNI 192.168.201.136@tcp132 [15447.560274] LNet: Removed LNI 192.168.201.136@tcp133 [15447.568464] LNet: Removed LNI 192.168.201.136@tcp134 [15447.584582] LNet: Removed LNI 192.168.201.136@tcp135 [15448.748323] Key type .llcrypt unregistered [15448.755847] Key type ._llcrypt unregistered [15476.512862] Key type ._llcrypt registered [15476.516548] Key type .llcrypt registered [15476.667595] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [15494.591792] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [15496.309465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15496.339414] alg: No test for adler32 (adler32-zlib) [15497.517156] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [15497.761066] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [15499.415413] Key type lgssc registered [15500.602443] Lustre: Echo OBD driver; http://www.lustre.org/ [15540.914758] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:11:45 (1787595105) [15563.739588] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [15566.907849] Key type lgssc unregistered [15567.173760] LNet: 472450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15567.190865] LNetError: 472450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15567.212433] LNet: Removed LNI 192.168.201.136@tcp [15568.013660] Key type .llcrypt unregistered [15568.016843] Key type ._llcrypt unregistered [15586.258703] Key type ._llcrypt registered [15586.260839] Key type .llcrypt registered [15586.368531] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [15587.170107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15587.225583] alg: No test for adler32 (adler32-zlib) [15588.424721] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [15588.757695] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [15590.431225] Key type lgssc registered [15591.574504] Lustre: Echo OBD driver; http://www.lustre.org/ [15600.983996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15601.024345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15602.364590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15602.393602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15602.491499] Lustre: lustre-MDT0000: new disk, initializing [15602.631538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15602.661877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15607.900369] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15620.066291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15620.208066] Lustre: 473663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15620.313995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15620.320394] Lustre: Skipped 1 previous similar message [15620.407836] Lustre: lustre-MDT0001: new disk, initializing [15620.526735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15620.562686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15620.575167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15625.706581] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15636.942685] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15643.846897] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15651.743291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15652.016755] Lustre: lustre-OST0000: new disk, initializing [15652.019687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15652.026486] Lustre: 475633:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15652.149841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15658.563612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15658.569381] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15658.714514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15659.550384] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15672.767875] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15678.102370] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing lnet_if_list [15684.396800] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15684.434704] LNet: Added LNI 192.168.201.136@tcp10 [8/256/0/180] [15691.077650] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15691.132139] LNet: Added LNI 192.168.201.136@tcp11 [8/256/0/180] [15696.107626] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15696.124747] LNet: Added LNI 192.168.201.136@tcp12 [8/256/0/180] [15701.113547] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15701.179868] LNet: Added LNI 192.168.201.136@tcp13 [8/256/0/180] [15707.539755] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15707.593197] LNet: Added LNI 192.168.201.136@tcp14 [8/256/0/180] [15713.539759] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15713.577137] LNet: Added LNI 192.168.201.136@tcp15 [8/256/0/180] [15720.853478] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15720.907858] LNet: Added LNI 192.168.201.136@tcp16 [8/256/0/180] [15727.978126] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15728.056910] LNet: Added LNI 192.168.201.136@tcp17 [8/256/0/180] [15733.759324] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15733.777778] LNet: Added LNI 192.168.201.136@tcp18 [8/256/0/180] [15739.859863] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15739.910234] LNet: Added LNI 192.168.201.136@tcp19 [8/256/0/180] [15745.872135] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15745.911545] LNet: Added LNI 192.168.201.136@tcp20 [8/256/0/180] [15751.253408] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15751.287265] LNet: Added LNI 192.168.201.136@tcp21 [8/256/0/180] [15757.103714] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15757.127581] LNet: Added LNI 192.168.201.136@tcp22 [8/256/0/180] [15762.561415] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15762.587599] LNet: Added LNI 192.168.201.136@tcp23 [8/256/0/180] [15767.696493] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15767.721565] LNet: Added LNI 192.168.201.136@tcp24 [8/256/0/180] [15773.447213] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15773.472736] LNet: Added LNI 192.168.201.136@tcp25 [8/256/0/180] [15779.588141] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15779.611280] LNet: Added LNI 192.168.201.136@tcp26 [8/256/0/180] [15786.265163] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15786.329827] LNet: Added LNI 192.168.201.136@tcp27 [8/256/0/180] [15794.661419] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15794.694084] LNet: Added LNI 192.168.201.136@tcp28 [8/256/0/180] [15802.502469] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15802.549129] LNet: Added LNI 192.168.201.136@tcp29 [8/256/0/180] [15809.935812] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15809.959879] LNet: Added LNI 192.168.201.136@tcp30 [8/256/0/180] [15816.238492] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15816.269505] LNet: Added LNI 192.168.201.136@tcp31 [8/256/0/180] [15823.055252] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15823.077479] LNet: Added LNI 192.168.201.136@tcp32 [8/256/0/180] [15829.928737] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15829.956143] LNet: Added LNI 192.168.201.136@tcp33 [8/256/0/180] [15838.204534] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15838.238244] LNet: Added LNI 192.168.201.136@tcp34 [8/256/0/180] [15847.030285] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15847.076418] LNet: Added LNI 192.168.201.136@tcp35 [8/256/0/180] [15854.162254] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15854.193870] LNet: Added LNI 192.168.201.136@tcp36 [8/256/0/180] [15861.724468] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15861.753515] LNet: Added LNI 192.168.201.136@tcp37 [8/256/0/180] [15869.590628] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15869.624596] LNet: Added LNI 192.168.201.136@tcp38 [8/256/0/180] [15877.777645] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15877.837341] LNet: Added LNI 192.168.201.136@tcp39 [8/256/0/180] [15886.541910] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15886.565756] LNet: Added LNI 192.168.201.136@tcp40 [8/256/0/180] [15895.443892] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15895.489833] LNet: Added LNI 192.168.201.136@tcp41 [8/256/0/180] [15902.980382] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15903.013993] LNet: Added LNI 192.168.201.136@tcp42 [8/256/0/180] [15911.343982] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15911.372326] LNet: Added LNI 192.168.201.136@tcp43 [8/256/0/180] [15918.902103] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15918.952106] LNet: Added LNI 192.168.201.136@tcp44 [8/256/0/180] [15927.837951] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15927.871979] LNet: Added LNI 192.168.201.136@tcp45 [8/256/0/180] [15935.740090] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15935.788287] LNet: Added LNI 192.168.201.136@tcp46 [8/256/0/180] [15944.016395] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15944.042034] LNet: Added LNI 192.168.201.136@tcp47 [8/256/0/180] [15952.438912] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15952.469038] LNet: Added LNI 192.168.201.136@tcp48 [8/256/0/180] [15960.871600] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15960.915831] LNet: Added LNI 192.168.201.136@tcp49 [8/256/0/180] [15969.813108] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15969.859142] LNet: Added LNI 192.168.201.136@tcp50 [8/256/0/180] [15991.777860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15991.781646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15991.796826] Lustre: Skipped 1 previous similar message [15991.827143] Lustre: Skipped 2 previous similar messages [15995.804441] Lustre: server umount lustre-MDT0000 complete [16000.131526] LustreError: 475645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595566 with bad export cookie 16659594370272171454 [16000.132733] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16000.138777] LustreError: 475645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16000.562464] Lustre: server umount lustre-MDT0001 complete [16018.271157] Lustre: 472937:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787595568/real 1787595568] req@0000000086f50cf4 x1874429377435520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787595584 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16018.299476] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16018.315018] Lustre: Skipped 1 previous similar message [16021.473297] Lustre: 472936:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787595571/real 1787595571] req@000000005cf9bc9d x1874429377435648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787595587 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16023.361596] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [16023.519128] Lustre: 472935:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787595573/real 1787595573] req@0000000009817a38 x1874429377435776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787595589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16026.465131] Lustre: 472937:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787595576/real 1787595576] req@0000000092b13d91 x1874429377436032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787595592 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16030.689101] Lustre: 472938:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787595581/real 1787595581] req@0000000001e5d924 x1874429377436160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787595597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16034.309563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16045.025766] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe732c4d03e2ba068 [16045.041763] Lustre: MGC192.168.201.136@tcp: Connection restored to 0@lo (at 0@lo) [16045.308032] LustreError: 475625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16045.333097] LustreError: 475625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [16045.381385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16047.399524] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16047.405895] LustreError: 475626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16051.511990] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [16052.708723] LustreError: 475625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16057.837241] LustreError: 475626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16061.208586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16061.586475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16063.674556] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16066.666500] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [16076.082193] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16082.379315] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16110.054226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16110.093214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16110.102455] Lustre: Skipped 1 previous similar message [16115.177988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16115.191694] Lustre: Skipped 1 previous similar message [16115.825957] Lustre: server umount lustre-OST0000 complete [16120.291368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16120.307835] Lustre: Skipped 1 previous similar message [16120.314658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16120.324584] Lustre: Skipped 1 previous similar message [16125.413870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16125.421358] Lustre: Skipped 1 previous similar message [16126.747337] Lustre: server umount lustre-MDT0000 complete [16130.536936] LustreError: 499884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16130.565573] LustreError: 499884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16131.208461] LustreError: 475645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595697 with bad export cookie 16659594370272174184 [16131.212436] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16131.217243] LustreError: 475645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16131.568522] Lustre: server umount lustre-MDT0001 complete [16145.190277] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [16149.327320] Key type lgssc unregistered [16149.875972] LNet: 502686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16149.888426] LNetError: 502686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16149.902781] LNet: Removed LNI 192.168.201.136@tcp [16149.909528] LNet: Removed LNI 192.168.201.136@tcp10 [16149.912445] LNet: Removed LNI 192.168.201.136@tcp11 [16149.919201] LNet: Removed LNI 192.168.201.136@tcp12 [16149.923371] LNet: Removed LNI 192.168.201.136@tcp13 [16149.931569] LNet: Removed LNI 192.168.201.136@tcp14 [16149.938431] LNet: Removed LNI 192.168.201.136@tcp15 [16149.944421] LNet: Removed LNI 192.168.201.136@tcp16 [16149.946424] LNet: Removed LNI 192.168.201.136@tcp17 [16149.951871] LNet: Removed LNI 192.168.201.136@tcp18 [16149.963370] LNet: Removed LNI 192.168.201.136@tcp19 [16149.967481] LNet: Removed LNI 192.168.201.136@tcp20 [16149.973961] LNet: Removed LNI 192.168.201.136@tcp21 [16149.977711] LNet: Removed LNI 192.168.201.136@tcp22 [16149.984518] LNet: Removed LNI 192.168.201.136@tcp23 [16149.987904] LNet: Removed LNI 192.168.201.136@tcp24 [16149.995377] LNet: Removed LNI 192.168.201.136@tcp25 [16149.998473] LNet: Removed LNI 192.168.201.136@tcp26 [16150.008912] LNet: Removed LNI 192.168.201.136@tcp27 [16150.013356] LNet: Removed LNI 192.168.201.136@tcp28 [16150.014665] LNet: Removed LNI 192.168.201.136@tcp29 [16150.024161] LNet: Removed LNI 192.168.201.136@tcp30 [16150.026207] LNet: Removed LNI 192.168.201.136@tcp31 [16150.041923] LNet: Removed LNI 192.168.201.136@tcp32 [16150.045840] LNet: Removed LNI 192.168.201.136@tcp33 [16150.053944] LNet: Removed LNI 192.168.201.136@tcp34 [16150.058182] LNet: Removed LNI 192.168.201.136@tcp35 [16150.060956] LNet: Removed LNI 192.168.201.136@tcp36 [16150.067356] LNet: Removed LNI 192.168.201.136@tcp37 [16150.078061] LNet: Removed LNI 192.168.201.136@tcp38 [16150.079831] LNet: Removed LNI 192.168.201.136@tcp39 [16150.085455] LNet: Removed LNI 192.168.201.136@tcp40 [16150.100983] LNet: Removed LNI 192.168.201.136@tcp41 [16150.107211] LNet: Removed LNI 192.168.201.136@tcp42 [16150.112124] LNet: Removed LNI 192.168.201.136@tcp43 [16150.115240] LNet: Removed LNI 192.168.201.136@tcp44 [16150.120892] LNet: Removed LNI 192.168.201.136@tcp45 [16150.128329] LNet: Removed LNI 192.168.201.136@tcp46 [16150.132813] LNet: Removed LNI 192.168.201.136@tcp47 [16150.143139] LNet: Removed LNI 192.168.201.136@tcp48 [16150.153792] LNet: Removed LNI 192.168.201.136@tcp49 [16150.180244] LNet: Removed LNI 192.168.201.136@tcp50 [16151.323801] Key type .llcrypt unregistered [16151.329552] Key type ._llcrypt unregistered [16175.277472] Key type ._llcrypt registered [16175.279820] Key type .llcrypt registered [16175.398845] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [16193.195951] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [16194.743065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16194.823595] alg: No test for adler32 (adler32-zlib) [16195.973696] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [16196.286714] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [16197.975623] Key type lgssc registered [16199.077885] Lustre: Echo OBD driver; http://www.lustre.org/ [16234.529962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16234.575835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16235.816391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16235.844546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16235.937295] Lustre: lustre-MDT0000: new disk, initializing [16236.023760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16236.041502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16240.538818] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [16251.064540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16251.293324] Lustre: 505888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16251.354787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16251.365870] Lustre: Skipped 1 previous similar message [16251.435584] Lustre: lustre-MDT0001: new disk, initializing [16251.538345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16251.577086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16251.596380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16256.392492] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [16267.921080] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16275.951246] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16285.610747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16285.907693] Lustre: *** cfs_fail_loc=90f, val=0*** [16286.001142] Lustre: lustre-OST0000: new disk, initializing [16286.007962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16286.021642] Lustre: 507903:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16286.140703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16291.954520] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [16295.530846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16295.549988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16295.647659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16302.576850] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16316.387688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16316.389739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16316.394887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16316.435062] Lustre: Skipped 1 previous similar message [16318.160986] Lustre: server umount lustre-OST0000 complete [16323.554551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16323.565328] LustreError: Skipped 1 previous similar message [16323.572323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16323.592204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16323.600819] Lustre: Skipped 1 previous similar message [16326.627239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16326.629620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16328.821427] Lustre: server umount lustre-MDT0000 complete [16332.654084] LustreError: 505879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595898 with bad export cookie 13876165829572162017 [16332.669260] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16333.063847] Lustre: server umount lustre-MDT0001 complete [16347.799446] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [16350.870977] Key type lgssc unregistered [16351.175538] LNet: 509813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16351.195775] LNetError: 509813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16352.230295] LNet: Removed LNI 192.168.201.136@tcp [16353.142112] Key type .llcrypt unregistered [16353.147987] Key type ._llcrypt unregistered [16376.161210] Key type ._llcrypt registered [16376.164982] Key type .llcrypt registered [16376.303285] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [16377.289305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16377.339940] alg: No test for adler32 (adler32-zlib) [16378.627912] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [16378.918347] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [16380.623140] Key type lgssc registered [16381.822020] Lustre: Echo OBD driver; http://www.lustre.org/ [16390.235892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16390.260970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16392.064821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16397.347981] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [16402.401657] LustreError: 511086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16402.441041] LustreError: 511086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16407.554879] LustreError: 511085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16408.049366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16408.510268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16413.577507] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [16422.145080] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16429.536108] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16438.054350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16438.351960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16445.937438] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [16454.481246] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16476.307621] loop0: detected capacity change from 0 to 204800000 [16476.482726] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16476.500745] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16476.525076] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16482.859679] loop0: detected capacity change from 0 to 204800000 [16483.154832] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16483.174375] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16483.206564] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16490.130179] loop0: detected capacity change from 0 to 204800000 [16496.222134] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16496.484575] Lustre: fs15246-OST0000: new disk, initializing [16496.488383] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16496.505582] Lustre: 514698:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16496.555927] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16503.482947] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [16513.777785] loop1: detected capacity change from 0 to 204800000 [16520.663108] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16520.772487] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16520.818508] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16520.878727] Lustre: fs15246-MDT0000: new disk, initializing [16521.047403] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16521.093110] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16521.305458] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16522.341085] Lustre: 515824:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000044f00314 x1874430205679488/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787596104 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16527.258715] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [16531.435485] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16531.454709] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16531.496176] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16554.489269] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16554.522234] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16559.599487] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16559.786110] Lustre: server umount fs15246-MDT0000 complete [16564.714962] LustreError: 513527:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16569.347744] Lustre: server umount fs15246-OST0000 complete [16585.185277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16585.195858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16585.206258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16585.227471] Lustre: Skipped 1 previous similar message [16590.281143] Lustre: server umount lustre-OST0000 complete [16595.424822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16595.428877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16595.454916] Lustre: Skipped 1 previous similar message [16595.475225] Lustre: Skipped 2 previous similar messages [16600.545537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16600.549513] Lustre: Skipped 1 previous similar message [16609.247235] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16609.403828] Lustre: server umount lustre-MDT0000 complete [16610.784775] LustreError: 511813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16610.805818] LustreError: 511813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16613.223109] LustreError: 511067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787596179 with bad export cookie 8309843871449070676 [16613.227335] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16613.237337] LustreError: 511067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16613.641123] Lustre: server umount lustre-MDT0001 complete [16628.273878] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [16632.837516] Key type lgssc unregistered [16633.323193] LNet: 518927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16633.344582] LNetError: 518927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16633.396441] LNet: Removed LNI 192.168.201.136@tcp [16634.638554] Key type .llcrypt unregistered [16634.640478] Key type ._llcrypt unregistered [16658.621374] Key type ._llcrypt registered [16658.623333] Key type .llcrypt registered [16658.748688] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [16659.760343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16660.033196] alg: No test for adler32 (adler32-zlib) [16661.378146] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [16661.713722] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [16663.567160] Key type lgssc registered [16664.892502] Lustre: Echo OBD driver; http://www.lustre.org/ [16708.704993] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [16721.335510] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [16760.750297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16760.788589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16762.190215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16762.244324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16762.359707] Lustre: lustre-MDT0000: new disk, initializing [16762.488609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16762.507479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16768.104684] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [16780.144577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16780.275159] Lustre: 523397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16780.359630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16780.363434] Lustre: Skipped 1 previous similar message [16780.435718] Lustre: lustre-MDT0001: new disk, initializing [16780.540390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16780.632152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16780.647199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16785.440539] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [16797.907952] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16804.461664] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16812.876236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16813.244455] Lustre: lustre-OST0000: new disk, initializing [16813.249578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16813.259174] Lustre: 525367:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16813.416488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16818.813124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16818.826099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16818.976205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16820.780084] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [16833.009382] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16838.628456] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16838.903277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16844.680405] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16844.904196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16849.895480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16849.901406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16849.910039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16849.920202] Lustre: Skipped 1 previous similar message [16849.935776] Lustre: Skipped 1 previous similar message [16855.011256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16855.018523] Lustre: Skipped 1 previous similar message [16860.136924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16860.142662] Lustre: Skipped 1 previous similar message [16861.151681] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16861.321105] Lustre: server umount lustre-OST0000 complete [16865.259997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16865.269584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16865.282603] Lustre: Skipped 1 previous similar message [16865.303492] Lustre: Skipped 1 previous similar message [16870.368788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16870.378986] Lustre: Skipped 1 previous similar message [16871.451717] Lustre: server umount lustre-MDT0000 complete [16875.619282] LustreError: 523388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787596441 with bad export cookie 11200526593774550214 [16875.619712] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16875.639911] LustreError: 523388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16876.039974] Lustre: server umount lustre-MDT0001 complete [16904.562088] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [16917.988440] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [16954.216418] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16640 sec ========= 14:35:18 (1787596518) [16956.240553] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:35:20 (1787596520) === [16960.812776] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:35:25 (1787596525) === [17006.648637] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [17009.304838] Key type lgssc unregistered [17009.526619] LNet: 532018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17009.531537] LNetError: 532018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17009.542388] LNet: Removed LNI 192.168.201.136@tcp [17010.356561] Key type .llcrypt unregistered [17010.358437] Key type ._llcrypt unregistered