[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 525206128 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003241] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007919] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010139] LSM: Security Framework initializing [ 0.011061] Yama: becoming mindful. [ 0.012044] SELinux: Initializing. [ 0.013076] *** VALIDATE selinux *** [ 0.021421] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026634] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028114] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030033] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031126] *** VALIDATE tmpfs *** [ 0.032491] *** VALIDATE proc *** [ 0.033273] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035275] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037134] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041023] Spectre V2 : User space: Vulnerable [ 0.042011] Speculative Store Bypass: Vulnerable [ 0.044933] debug: unmapping init [mem 0xffffffffa1259000-0xffffffffa1260fff] [ 0.046162] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047711] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048023] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051016] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053017] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055249] rcu: Hierarchical SRCU implementation. [ 0.057305] smp: Bringing up secondary CPUs ... [ 0.058584] x86: Booting SMP configuration: [ 0.059034] .... node #0, CPUs: #1 #2 #3 [ 0.064676] smp: Brought up 1 node, 4 CPUs [ 0.066011] smpboot: Max logical packages: 1 [ 0.067012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130322] node 0 deferred pages initialised in 60ms [ 0.133118] devtmpfs: initialized [ 0.134228] x86/mm: Memory block size: 128MB [ 0.136685] gcov: version magic: 0x41383552 [ 0.138386] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143252] pinctrl core: initialized pinctrl subsystem [ 0.144171] [ 0.144669] ************************************************************* [ 0.145013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146013] ** ** [ 0.147012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.148011] ** ** [ 0.149016] ** This means that this kernel is built to expose internal ** [ 0.150012] ** IOMMU data structures, which may compromise security on ** [ 0.151013] ** your system. ** [ 0.152013] ** ** [ 0.153013] ** If you see this message and you are not debugging the ** [ 0.154014] ** kernel, report this immediately to your vendor! ** [ 0.155013] ** ** [ 0.156012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157013] ************************************************************* [ 0.158694] NET: Registered protocol family 16 [ 0.159458] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.160056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.161061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.162418] cpuidle: using governor menu [ 0.163179] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.164000] PCI: Using configuration type 1 for base access [ 0.166126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.177154] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.179020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.184080] cryptd: max_cpu_qlen set to 1000 [ 0.189259] ACPI: Added _OSI(Module Device) [ 0.191014] ACPI: Added _OSI(Processor Device) [ 0.193012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200894] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.208045] ACPI: Interpreter enabled [ 0.209069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210013] ACPI: Using IOAPIC for interrupt routing [ 0.212111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.215377] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.226896] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.229039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.231016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.239061] acpiphp: Slot [2] registered [ 0.240096] acpiphp: Slot [5] registered [ 0.242114] acpiphp: Slot [6] registered [ 0.243081] acpiphp: Slot [7] registered [ 0.244106] acpiphp: Slot [8] registered [ 0.245097] acpiphp: Slot [9] registered [ 0.246182] acpiphp: Slot [10] registered [ 0.248120] acpiphp: Slot [3] registered [ 0.249096] acpiphp: Slot [4] registered [ 0.250095] acpiphp: Slot [11] registered [ 0.252109] acpiphp: Slot [12] registered [ 0.254127] acpiphp: Slot [13] registered [ 0.255100] acpiphp: Slot [14] registered [ 0.257113] acpiphp: Slot [15] registered [ 0.258188] acpiphp: Slot [16] registered [ 0.260138] acpiphp: Slot [17] registered [ 0.262101] acpiphp: Slot [18] registered [ 0.264126] acpiphp: Slot [19] registered [ 0.265106] acpiphp: Slot [20] registered [ 0.267111] acpiphp: Slot [21] registered [ 0.269104] acpiphp: Slot [22] registered [ 0.270219] acpiphp: Slot [23] registered [ 0.272083] acpiphp: Slot [24] registered [ 0.274123] acpiphp: Slot [25] registered [ 0.275097] acpiphp: Slot [26] registered [ 0.277106] acpiphp: Slot [27] registered [ 0.278114] acpiphp: Slot [28] registered [ 0.280111] acpiphp: Slot [29] registered [ 0.281101] acpiphp: Slot [30] registered [ 0.282063] acpiphp: Slot [31] registered [ 0.284055] PCI host bridge to bus 0000:00 [ 0.285018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.287028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.289022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.291022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.293021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.296021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.297230] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.299946] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.303256] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.312014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.317059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.319019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.321015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.323022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.325450] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.330070] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.334044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.336940] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.343018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.358015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.364018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.372010] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.386017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.399096] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.434016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.447583] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.476021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.502018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.519991] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.531021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.541046] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.560019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.583773] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.600024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.613022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.650018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.669072] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.679033] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.689021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.714021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.728290] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.750019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.761017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.776036] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.789940] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.792491] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.795448] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.797601] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.800319] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.806408] iommu: Default domain type: Passthrough [ 0.808761] SCSI subsystem initialized [ 0.810316] ACPI: bus type USB registered [ 0.812319] usbcore: registered new interface driver usbfs [ 0.815153] usbcore: registered new interface driver hub [ 0.817215] usbcore: registered new device driver usb [ 0.819250] pps_core: LinuxPPS API ver. 1 registered [ 0.821019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.825091] PTP clock support registered [ 0.828227] EDAC MC: Ver: 3.0.0 [ 0.830017] PCI: Using ACPI for IRQ routing [ 0.832142] NetLabel: Initializing [ 0.833012] NetLabel: domain hash size = 128 [ 0.835012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.837105] NetLabel: unlabeled traffic allowed by default [ 0.840169] vgaarb: loaded [ 0.842360] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.844024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.853000] clocksource: Switched to clocksource kvm-clock [ 0.978562] VFS: Disk quotas dquot_6.6.0 [ 0.980845] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.983814] *** VALIDATE ramfs *** [ 0.985393] *** VALIDATE hugetlbfs *** [ 0.987322] pnp: PnP ACPI init [ 0.990268] pnp: PnP ACPI: found 6 devices [ 1.006393] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.009638] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.011577] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.013816] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.016476] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.018831] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.021101] NET: Registered protocol family 2 [ 1.023520] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.028114] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.031221] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.036583] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.039669] TCP: Hash tables configured (established 65536 bind 65536) [ 1.042617] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.045829] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.048573] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.051150] NET: Registered protocol family 1 [ 1.054947] RPC: Registered named UNIX socket transport module. [ 1.057321] RPC: Registered udp transport module. [ 1.059485] RPC: Registered tcp transport module. [ 1.060849] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.062721] NET: Registered protocol family 44 [ 1.064273] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.065780] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.067508] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.069711] PCI: CLS 0 bytes, default 64 [ 1.071255] Unpacking initramfs... [ 2.544260] debug: unmapping init [mem 0xffff9feb3cc54000-0xffff9feb3ffbffff] [ 2.548402] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.550829] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.553973] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.040244] Initialise system trusted keyrings [ 3.041912] Key type blacklist registered [ 3.043610] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.053132] zbud: loaded [ 3.055921] *** VALIDATE nfs *** [ 3.057268] *** VALIDATE nfs4 *** [ 3.058845] pstore: using deflate compression [ 3.062201] Platform Keyring initialized [ 3.152737] NET: Registered protocol family 38 [ 3.154785] Key type asymmetric registered [ 3.156916] Asymmetric key parser 'x509' registered [ 3.159814] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.164301] io scheduler mq-deadline registered [ 3.166369] io scheduler kyber registered [ 3.168196] io scheduler bfq registered [ 3.170553] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.175833] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.178967] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.181990] ACPI: Power Button [PWRF] [ 3.188951] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.196564] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.213699] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.224993] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.242740] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.269420] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.297168] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.301716] Non-volatile memory driver v1.3 [ 3.302937] Linux agpgart interface v0.103 [ 3.335297] virtio_blk virtio1: [vda] 146712 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.338341] vda: detected capacity change from 0 to 75116544 [ 3.365045] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.367622] vdb: detected capacity change from 0 to 1073741824 [ 3.385347] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.389510] vdc: detected capacity change from 0 to 2621440000 [ 3.407686] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.411588] vdd: detected capacity change from 0 to 2621440000 [ 3.430794] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.434247] vde: detected capacity change from 0 to 4294967296 [ 3.456689] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.458946] vdf: detected capacity change from 0 to 4294967296 [ 3.480936] libphy: Fixed MDIO Bus: probed [ 3.494970] usbcore: registered new interface driver usbserial_generic [ 3.499452] usbserial: USB Serial support registered for generic [ 3.502166] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.506619] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.508607] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.511390] mousedev: PS/2 mouse device common for all mice [ 3.514664] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.516668] rtc_cmos 00:05: RTC can wake from S4 [ 3.520642] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.523087] rtc_cmos 00:05: registered as rtc0 [ 3.526703] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.527017] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.532383] intel_pstate: CPU model not supported [ 3.534553] hid: raw HID events driver (C) Jiri Kosina [ 3.536343] usbcore: registered new interface driver usbhid [ 3.538306] usbhid: USB HID core driver [ 3.539848] drop_monitor: Initializing network drop monitor service [ 3.542311] Initializing XFRM netlink socket [ 3.544356] NET: Registered protocol family 10 [ 3.547559] Segment Routing with IPv6 [ 3.548978] NET: Registered protocol family 17 [ 3.551022] mpls_gso: MPLS GSO support [ 3.556785] RAS: Correctable Errors collector initialized. [ 3.558175] AVX version of gcm_enc/dec engaged. [ 3.559080] AES CTR mode by8 optimization enabled [ 3.626984] sched_clock: Marking stable (3626903407, 0)->(4553872753, -926969346) [ 3.630667] registered taskstats version 1 [ 3.633246] Loading compiled-in X.509 certificates [ 3.635273] zswap: loaded using pool lzo/zbud [ 3.657614] Key type big_key registered [ 3.668926] Key type encrypted registered [ 3.670120] ima: No TPM chip found, activating TPM-bypass! [ 3.671674] ima: Allocated hash algorithm: sha1 [ 3.673483] ima: No architecture policies found [ 3.674557] evm: Initialising EVM extended attributes: [ 3.676295] evm: security.selinux [ 3.677839] evm: security.ima [ 3.679020] evm: security.capability [ 3.680538] evm: HMAC attrs: 0x1 [ 3.683330] rtc_cmos 00:05: setting system clock to 2026-09-08 04:52:53 UTC (1788843173) [ 3.691536] debug: unmapping init [mem 0xffffffffa2203000-0xffffffffa23fffff] [ 3.694832] debug: unmapping init [mem 0xffffffffa0f82000-0xffffffffa1258fff] [ 3.702190] Write protecting the kernel read-only data: 28672k [ 3.705361] debug: unmapping init [mem 0xffffffff9f603000-0xffffffff9f7fffff] [ 3.707422] debug: unmapping init [mem 0xffffffff9ff14000-0xffffffff9fffffff] [ 3.738350] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.746363] systemd[1]: Detected virtualization kvm. [ 3.748430] systemd[1]: Detected architecture x86-64. [ 3.749614] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.774276] systemd[1]: No hostname configured. [ 3.775981] systemd[1]: Set hostname to . [ 3.778085] random: systemd: uninitialized urandom read (16 bytes read) [ 3.780580] systemd[1]: Initializing machine ID from random generator. [ 3.900878] random: systemd: uninitialized urandom read (16 bytes read) [ 3.903405] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.908943] random: systemd: uninitialized urandom read (16 bytes read) [ 3.911707] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.917041] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.553200] device-mapper: uevent: version 1.0.3 [ 4.554989] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK [ 5.277717] random: fast init done ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.373857] virtio_net virtio0 ens2: renamed from eth0 [ 5.457817] scsi host0: ata_piix [ 5.474942] scsi host1: ata_piix [ 5.476671] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.479475] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.921214] dracut-initqueue[592]: RTNETLINK answers: File exists [ 10.186193] random: crng init done [ 10.188067] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK [0[ 11.135812] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) m] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.006673] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.018984] SELinux: Disabled at runtime. [ 16.189377] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.210847] systemd[1]: Detected virtualization kvm. [ 16.214579] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.908867] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.913629] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.928238] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.937161] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.942886] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.960492] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.970294] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch.[ 18.479609] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Journal Service. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 19.895087] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.272503] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.629684] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.355031] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.607407] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** [ 27.495813] Key type dns_resolver registered ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 28.743739] NFS: Registering the id_resolver key type [ 28.751818] Key type id_resolver registered [ 28.757184] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg210-server login: [ 106.259735] libcfs: loading out-of-tree module taints kernel. [ 106.360694] Key type ._llcrypt registered [ 106.364453] Key type .llcrypt registered [ 106.511634] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 130.213287] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 132.176939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 132.194909] alg: No test for adler32 (adler32-zlib) [ 133.614584] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 134.651539] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 136.479709] Key type lgssc registered [ 138.907803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 159.834769] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 164.622036] hrtimer: interrupt took 4314923 ns [ 212.314522] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 229.849476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 229.891477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 231.300858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 231.342808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 231.437759] Lustre: lustre-MDT0000: new disk, initializing [ 231.490751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.541052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 236.799713] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.300841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 252.448729] Lustre: 6536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 252.484773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 252.489245] Lustre: Skipped 1 previous similar message [ 252.625176] Lustre: lustre-MDT0001: new disk, initializing [ 252.710485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 252.755272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 252.776614] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 258.094977] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.513819] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 274.551124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 274.923659] Lustre: lustre-OST0000: new disk, initializing [ 274.938835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 274.950612] Lustre: 8476:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 275.045533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 282.945163] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 284.734428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 284.749832] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 284.855703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 299.738683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 299.914402] Lustre: lustre-OST0001: new disk, initializing [ 299.922381] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 299.931924] Lustre: 9550:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 300.036752] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 307.093295] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 309.332052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 309.352153] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 309.440569] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 320.021758] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 332.522689] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 339.954559] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing check_logdir /tmp/testlogs/ [ 345.341612] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing yml_node [ 351.860911] Lustre: DEBUG MARKER: Client: 2.17.57.111 [ 355.413555] Lustre: DEBUG MARKER: MDS: 2.17.57.111 [ 358.621289] Lustre: DEBUG MARKER: OSS: 2.17.57.111 [ 360.851494] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 00:58:48 EDT 2026 [ 381.945960] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 384.023364] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 401.375957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 401.390352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.408377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 401.890700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.917275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 401.940911] Lustre: Skipped 2 previous similar messages [ 409.038696] Lustre: server umount lustre-MDT0000 complete [ 412.129225] LustreError: 6546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.148603] LustreError: 6546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 413.673361] LustreError: 7484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843583 with bad export cookie 4345659553131460844 [ 413.674462] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 413.678980] LustreError: 7484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 417.248068] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 417.252030] LustreError: 6541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 417.252202] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 417.259857] Lustre: Skipped 3 previous similar messages [ 417.315636] LustreError: 6541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 418.272137] LustreError: 6543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 418.286671] LustreError: 6543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 420.065262] Lustre: server umount lustre-MDT0001 complete [ 424.493308] Lustre: server umount lustre-OST0000 complete [ 427.559655] Lustre: server umount lustre-OST0001 complete [ 434.325888] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 444.941799] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 486.238976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.583171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 486.633454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 486.720812] Lustre: lustre-MDT0000: new disk, initializing [ 486.802089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 486.828468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 491.265859] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 502.803346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.977671] Lustre: 15106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 503.080192] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 503.091494] Lustre: Skipped 1 previous similar message [ 503.197869] Lustre: lustre-MDT0001: new disk, initializing [ 503.360115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 503.443461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 503.461187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 510.772986] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 523.734160] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 530.495399] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 537.894305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 538.123530] Lustre: lustre-OST0000: new disk, initializing [ 538.127225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 538.131512] Lustre: 17088:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 538.279278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 539.613599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 539.634745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 539.800471] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 545.953518] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 557.170787] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 562.285917] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 562.546626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 567.797603] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 568.062581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 570.336257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 570.339637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.342713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 570.353911] Lustre: Skipped 1 previous similar message [ 570.365833] Lustre: Skipped 4 previous similar messages [ 575.474324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 576.424738] Lustre: server umount lustre-OST0000 complete [ 580.589595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.184463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 585.193819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 585.198948] Lustre: Skipped 2 previous similar messages [ 586.893750] Lustre: server umount lustre-MDT0000 complete [ 590.822101] LustreError: 15970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.861773] LustreError: 15970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 591.483960] LustreError: 15096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843761 with bad export cookie 4345659553131462377 [ 591.524625] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.996665] Lustre: server umount lustre-MDT0001 complete [ 596.998176] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:02:44 (1788843764) [ 599.993691] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 602.196707] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:02:49 (1788843769) [ 610.214708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.833704] LustreError: 19497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.889155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 616.557770] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 621.035507] LustreError: 19498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.058072] LustreError: 19498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 626.530497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 632.195171] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 641.456817] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 647.873974] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 654.760748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 662.976885] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 671.496362] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 681.952550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 681.956169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 681.959109] LustreError: Skipped 1 previous similar message [ 681.977675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 681.988057] Lustre: Skipped 2 previous similar messages [ 682.017690] Lustre: Skipped 2 previous similar messages [ 687.409974] Lustre: server umount lustre-OST0000 complete [ 693.754684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 698.020511] Lustre: server umount lustre-MDT0000 complete [ 700.387158] LustreError: 19497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 700.418382] LustreError: 19497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 702.321397] LustreError: 19479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843872 with bad export cookie 4345659553131463259 [ 702.328264] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 702.335611] LustreError: 19479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 702.845106] Lustre: server umount lustre-MDT0001 complete [ 717.004898] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 720.927635] Key type lgssc unregistered [ 721.224356] LNet: 23031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.236820] LNetError: 23031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.262187] LNet: Removed LNI 192.168.202.110@tcp [ 722.468064] Key type .llcrypt unregistered [ 722.474392] Key type ._llcrypt unregistered [ 749.222653] Key type ._llcrypt registered [ 749.224974] Key type .llcrypt registered [ 749.373384] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 751.312473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 751.351014] alg: No test for adler32 (adler32-zlib) [ 752.466562] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 752.782274] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 754.519408] Key type lgssc registered [ 755.645803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 763.914419] loop: module loaded [ 763.993921] loop0: detected capacity change from 0 to 204800000 [ 764.295725] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 764.302564] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 764.751912] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 771.735825] loop0: detected capacity change from 0 to 204800000 [ 772.097156] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 772.111820] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 772.120651] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 779.775824] loop0: detected capacity change from 0 to 204800000 [ 787.943593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 788.006503] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.512669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 789.650534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 789.778563] Lustre: lustre-MDT0000: new disk, initializing [ 789.885315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 789.931101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 794.582299] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 803.943869] loop1: detected capacity change from 0 to 204800000 [ 810.090246] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 810.398687] Lustre: lustre-OST07c6: new disk, initializing [ 810.403767] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 810.408272] Lustre: Skipped 1 previous similar message [ 810.414769] Lustre: 26216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 810.493343] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 817.762246] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 820.335496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 820.358655] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 820.395145] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 823.687372] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 823.695468] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 823.706153] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 823.719108] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 823.728061] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 823.735709] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 838.320150] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 838.357778] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 838.376977] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 838.394590] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 838.417878] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 838.441722] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 843.816797] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 843.828174] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 843.838180] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 843.865781] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 843.894916] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 843.915089] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 843.940520] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 843.955334] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 843.968064] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 843.987536] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 844.002901] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 844.019112] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 847.269410] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 847.286454] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 847.295251] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 847.304228] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 847.311261] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 847.319038] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 847.330257] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 847.342197] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 847.346995] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 847.352318] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 847.361079] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 847.367269] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 852.769597] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 852.780694] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 852.794466] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 852.801489] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 852.806027] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 852.813696] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 852.829436] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 852.850362] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 852.869335] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 852.882168] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 852.896801] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 852.905249] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 861.922361] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 861.939181] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 861.951165] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 861.966500] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 861.978439] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 861.987667] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 861.994039] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 862.006618] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 862.019811] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 862.029403] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 862.035810] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 862.042178] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 922.602768] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 922.616422] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.640409] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 924.726519] Lustre: server umount lustre-OST07c6 complete [ 936.448310] Lustre: server umount lustre-MDT0000 complete [ 954.832127] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 958.092967] Key type lgssc unregistered [ 958.407763] LNet: 28775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.417441] LNetError: 28775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.471366] LNet: Removed LNI 192.168.202.110@tcp [ 960.312145] Key type .llcrypt unregistered [ 960.316097] Key type ._llcrypt unregistered [ 983.262289] Key type ._llcrypt registered [ 983.264535] Key type .llcrypt registered [ 983.369644] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 984.334620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 984.371773] alg: No test for adler32 (adler32-zlib) [ 985.647290] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 985.970765] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 987.647170] Key type lgssc registered [ 989.046407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 997.909339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 997.947849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.526625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.316685] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1009.653992] LustreError: 30041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1009.681700] LustreError: 30041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1013.590235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.833497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1017.912350] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1026.281926] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1031.922810] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1039.005626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1039.417060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1044.523459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1045.173318] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1052.975835] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1074.145356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1074.154213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.161741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1074.178540] Lustre: Skipped 1 previous similar message [ 1077.781612] Lustre: server umount lustre-OST0000 complete [ 1084.391728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.403137] Lustre: Skipped 1 previous similar message [ 1084.403406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.410200] Lustre: Skipped 2 previous similar messages [ 1088.444207] Lustre: server umount lustre-MDT0000 complete [ 1089.508941] LustreError: 30036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.533364] LustreError: 30036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1092.642250] LustreError: 30020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844262 with bad export cookie 14103206272722319019 [ 1092.647892] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1092.652912] LustreError: 30020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1093.064988] Lustre: server umount lustre-MDT0001 complete [ 1104.874805] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1107.790895] Key type lgssc unregistered [ 1108.097592] LNet: 33777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.102197] LNetError: 33777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1108.130596] LNet: Removed LNI 192.168.202.110@tcp [ 1109.283802] Key type .llcrypt unregistered [ 1109.289359] Key type ._llcrypt unregistered [ 1129.573579] Key type ._llcrypt registered [ 1129.582429] Key type .llcrypt registered [ 1129.782654] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1130.866535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1130.888117] alg: No test for adler32 (adler32-zlib) [ 1131.987817] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1132.202124] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1133.839190] Key type lgssc registered [ 1134.744435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1141.158889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1141.179835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1142.639408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1147.245441] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1152.996979] LustreError: 35045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1153.021484] LustreError: 35045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1158.116977] LustreError: 35046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1159.149431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.838082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1164.268603] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1172.373600] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1178.254293] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1185.392868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1185.737506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1187.772254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1192.334443] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1200.843684] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1218.528225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1218.531498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1218.560636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1220.577862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1220.578415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.581065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1222.229638] Lustre: server umount lustre-OST0000 complete [ 1230.820987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.854874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1230.870974] Lustre: Skipped 2 previous similar messages [ 1232.723855] Lustre: server umount lustre-MDT0000 complete [ 1235.940544] LustreError: 35775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.957393] LustreError: 35775:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1236.549870] LustreError: 35782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844406 with bad export cookie 6532393453578787438 [ 1236.552511] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1236.570021] LustreError: 35782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1237.094983] Lustre: server umount lustre-MDT0001 complete [ 1247.887180] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1250.938312] Key type lgssc unregistered [ 1251.268896] LNet: 38599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.275285] LNetError: 38599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.288228] LNet: Removed LNI 192.168.202.110@tcp [ 1252.048837] Key type .llcrypt unregistered [ 1252.050823] Key type ._llcrypt unregistered [ 1273.956727] Key type ._llcrypt registered [ 1273.961273] Key type .llcrypt registered [ 1274.132564] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1275.141036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1275.384570] alg: No test for adler32 (adler32-zlib) [ 1276.512523] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1276.818523] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1278.519304] Key type lgssc registered [ 1279.917284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1288.070730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1288.091956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.775228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1294.839798] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1299.938238] LustreError: 39992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.954286] LustreError: 39992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1303.251198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.608324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1307.863558] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1315.049873] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1321.261722] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1330.008756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1330.299694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.406075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1339.479970] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1347.511197] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1354.153944] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 1354.211763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1354.230772] Lustre: Skipped 2 previous similar messages [ 1359.258576] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 1359.263437] Lustre: Skipped 3 previous similar messages [ 1360.135217] Lustre: server umount lustre-MDT0000 complete [ 1363.839794] LustreError: 39973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844533 with bad export cookie 14804489365471158938 [ 1363.855327] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1363.856593] LustreError: 39973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1364.369420] LustreError: 39988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.395412] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.10@tcp (stopping) [ 1364.399232] Lustre: Skipped 3 previous similar messages [ 1364.451234] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1368.482890] LustreError: 39986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1368.483764] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1368.508239] LustreError: 39986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1368.548128] Lustre: Skipped 1 previous similar message [ 1373.603937] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1373.605074] LustreError: 39988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1373.613752] Lustre: Skipped 1 previous similar message [ 1373.649890] LustreError: 39988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1378.274156] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1378.498266] Lustre: server umount lustre-MDT0001 complete [ 1393.990045] Lustre: server umount lustre-OST0000 complete [ 1411.985598] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1415.164478] Key type lgssc unregistered [ 1415.420867] LNet: 43851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1415.426452] LNetError: 43851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1415.441942] LNet: Removed LNI 192.168.202.110@tcp [ 1416.431786] Key type .llcrypt unregistered [ 1416.436313] Key type ._llcrypt unregistered [ 1438.797389] Key type ._llcrypt registered [ 1438.800672] Key type .llcrypt registered [ 1439.003472] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1440.696839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1440.729609] alg: No test for adler32 (adler32-zlib) [ 1441.975421] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1442.363154] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1444.224971] Key type lgssc registered [ 1446.311705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1455.977291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1456.054160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1457.924495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1462.944930] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1467.365311] LustreError: 45267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.404684] LustreError: 45267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1472.505709] LustreError: 45268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.232617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.805867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1478.941723] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1489.470950] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1497.259600] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1507.359178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1507.973732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1516.294145] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1526.346247] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1527.477324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1533.923551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1533.931555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1533.945675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1538.023701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1538.025464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1538.032308] Lustre: Skipped 1 previous similar message [ 1538.055900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.494319] Lustre: lustre-OST0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 1539.514430] Lustre: Skipped 1 previous similar message [ 1539.757814] Lustre: server umount lustre-OST0000 complete [ 1561.061038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1561.068659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1561.080738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1564.930669] Lustre: server umount lustre-MDT0000 complete [ 1568.736110] LustreError: 45268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.751943] LustreError: 45268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1569.773861] LustreError: 45247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844739 with bad export cookie 13341305896552773750 [ 1569.782430] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1569.789289] LustreError: 45247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1576.364844] Lustre: server umount lustre-MDT0001 complete [ 1589.427350] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1592.515511] Key type lgssc unregistered [ 1593.078049] LNet: 48977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.095689] LNetError: 48977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1593.140956] LNet: Removed LNI 192.168.202.110@tcp [ 1594.490502] Key type .llcrypt unregistered [ 1594.493078] Key type ._llcrypt unregistered [ 1624.044081] Key type ._llcrypt registered [ 1624.046014] Key type .llcrypt registered [ 1624.163577] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1625.151371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1625.339745] alg: No test for adler32 (adler32-zlib) [ 1626.335398] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1626.540833] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1628.207266] Key type lgssc registered [ 1629.908818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1640.039488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1640.059796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.713467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1646.681913] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1652.195307] LustreError: 50393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.225342] LustreError: 50393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1656.684450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.172967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1662.233734] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1670.382181] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1677.237384] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1684.668255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1684.937825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.057647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1691.890163] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1701.072196] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1706.835684] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1710.498940] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1723.507335] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-08 1h21m30s [ 1725.994509] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1728.924322] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 1728.995982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1729.011109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.953530] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 1729.969206] Lustre: Skipped 8 previous similar messages [ 1734.994798] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 1735.004926] Lustre: Skipped 3 previous similar messages [ 1741.201107] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 1741.224786] Lustre: Skipped 3 previous similar messages [ 1743.327076] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1744.194798] Lustre: server umount lustre-MDT0000 complete [ 1747.354159] LustreError: 50390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1748.971368] LustreError: 50394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1748.995845] LustreError: 50394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1749.726668] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1753.501340] LustreError: 50389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1756.814230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1756.899863] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1757.261779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.315894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1758.270895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1762.660585] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1767.210807] Lustre: DEBUG MARKER: Wait for df (36804) ... [ 1769.405396] Lustre: DEBUG MARKER: done [ 1777.639099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1777.649484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.679943] Lustre: Skipped 2 previous similar messages [ 1777.694900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1777.698672] Lustre: Skipped 3 previous similar messages [ 1780.840915] Lustre: server umount lustre-OST0000 complete [ 1788.897033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1788.902312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.913377] LustreError: Skipped 1 previous similar message [ 1788.917738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1788.927914] Lustre: Skipped 2 previous similar messages [ 1788.931444] Lustre: Skipped 2 previous similar messages [ 1792.186423] Lustre: server umount lustre-MDT0000 complete [ 1796.067046] LustreError: 50393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1796.080610] LustreError: 50393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 1796.802204] LustreError: 50374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844966 with bad export cookie 17089603265377252102 [ 1796.810369] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1796.818061] LustreError: 50374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1797.349437] Lustre: server umount lustre-MDT0001 complete [ 1810.406398] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1813.348933] Key type lgssc unregistered [ 1813.756843] LNet: 55322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.760959] LNetError: 55322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.788513] LNet: Removed LNI 192.168.202.110@tcp [ 1814.561362] Key type .llcrypt unregistered [ 1814.563437] Key type ._llcrypt unregistered [ 1857.980623] Key type ._llcrypt registered [ 1857.985193] Key type .llcrypt registered [ 1858.128124] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1859.320974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1859.355787] alg: No test for adler32 (adler32-zlib) [ 1860.375903] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1860.563777] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1862.183335] Key type lgssc registered [ 1863.031901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1870.645520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1870.676149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1871.972641] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1871.988580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1872.137467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1877.485809] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1887.180336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.265563] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1887.286898] Lustre: 57525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1887.452600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1892.035954] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1901.169199] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1908.198456] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1917.212796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1917.380364] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1917.672243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.289976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1926.286849] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1937.646679] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1945.244272] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-08 1h25m12s [ 1947.333305] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1955.679113] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1958.195493] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1958.519673] LustreError: 59835:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1978.822825] Lustre: lustre-MDT0000: Client 76cc937d-72fe-4bad-ad25-bff08c45985d (at 192.168.202.10@tcp) reconnecting [ 1998.551142] LustreError: 59835:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2001.792378] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2004.789284] Lustre: DEBUG MARKER: done [ 2015.720342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2015.724373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.726956] LustreError: Skipped 1 previous similar message [ 2015.732773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2015.743735] Lustre: Skipped 1 previous similar message [ 2015.765141] Lustre: Skipped 1 previous similar message [ 2017.162528] Lustre: server umount lustre-OST0000 complete [ 2025.954886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2025.956266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.967574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2025.982518] Lustre: Skipped 1 previous similar message [ 2025.992179] Lustre: Skipped 1 previous similar message [ 2027.048037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2027.058361] Lustre: Skipped 1 previous similar message [ 2027.213336] Lustre: server umount lustre-MDT0000 complete [ 2030.938195] LustreError: 57518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845200 with bad export cookie 16192240877043325782 [ 2030.941222] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2030.948109] LustreError: 57518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2031.282526] Lustre: server umount lustre-MDT0001 complete [ 2043.309242] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2046.324346] Key type lgssc unregistered [ 2046.690330] LNet: 61855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.699391] LNetError: 61855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.726778] LNet: Removed LNI 192.168.202.110@tcp [ 2047.554534] Key type .llcrypt unregistered [ 2047.556315] Key type ._llcrypt unregistered [ 2100.965486] Key type ._llcrypt registered [ 2100.967361] Key type .llcrypt registered [ 2101.091756] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2101.776633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2101.897880] alg: No test for adler32 (adler32-zlib) [ 2102.961191] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2103.142495] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2104.767172] Key type lgssc registered [ 2106.412932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2110.219390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2110.259152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.623949] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2111.662722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2111.947942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2113.149542] Lustre: Failing over lustre-MDT0000 [ 2113.787549] Lustre: server umount lustre-MDT0000 complete [ 2123.477911] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:28:10 (1788845290) [ 2132.466075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.012253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.748657] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2150.083182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2150.175059] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2150.239510] Lustre: 64651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2150.634736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2155.996743] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2164.730961] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2171.211950] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2178.973288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2179.225686] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2179.416876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.434365] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2189.563148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2196.299282] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2203.297873] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2212.320332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2212.323969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.341815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2215.396194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2215.400651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2215.402417] Lustre: Skipped 1 previous similar message [ 2215.420960] Lustre: Skipped 1 previous similar message [ 2218.133493] Lustre: server umount lustre-MDT0000 complete [ 2223.509986] LustreError: 64644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845393 with bad export cookie 17992899842915494930 [ 2223.518726] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2223.523632] LustreError: 64644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2223.992637] Lustre: server umount lustre-MDT0001 complete [ 2229.362525] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2235.654764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2242.023379] Lustre: 63575:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788845395/real 1788845395] req@00000000c485310e x1875740219974528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788845411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2242.066670] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.087140] Lustre: 63573:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788845398/real 1788845398] req@000000007bb0b4ec x1875740219974656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788845414 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2247.135723] Lustre: 63572:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788845400/real 1788845400] req@00000000ac22b202 x1875740219974784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788845416 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2248.070344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.184716] Lustre: 63574:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788845403/real 1788845403] req@0000000024db01b5 x1875740219975040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788845419 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2254.303561] Lustre: 63575:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788845408/real 1788845408] req@00000000be7b9952 x1875740219975168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788845424 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2268.706816] LustreError: 63571:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c485310e x1875740219975936/t0(0) o250->MGC192.168.202.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2269.145928] LustreError: 66503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.172703] LustreError: 66503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2269.271484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.371240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2270.310981] LustreError: 66504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.320908] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2271.345421] LustreError: 66503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.627723] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2276.838342] LustreError: 66504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.975697] LustreError: 66503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.047938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.542229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2286.637512] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2288.514595] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2297.814937] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2305.308584] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2312.166964] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2312.176901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2312.183789] Lustre: Skipped 1 previous similar message [ 2312.202482] Lustre: Skipped 2 previous similar messages [ 2314.436457] Lustre: server umount lustre-MDT0000 complete [ 2317.285635] LustreError: 68431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.312075] LustreError: 68431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2318.829088] LustreError: 65389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845488 with bad export cookie 17992899842915497205 [ 2318.830354] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2318.842784] LustreError: 65389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2319.280316] Lustre: server umount lustre-MDT0001 complete [ 2325.429218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2335.020576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2338.784462] Lustre: 63572:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788845492/real 1788845492] req@0000000060e8ed81 x1875740220003456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788845508 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2338.811675] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.821654] Lustre: Skipped 1 previous similar message [ 2343.967929] LustreError: 63571:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f58645d9 x1875740220004608/t0(0) o250->MGC192.168.202.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2344.328252] LustreError: 66503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.391691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2344.468609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2349.229366] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2358.213742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.877761] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2363.887694] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2368.037202] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2371.670685] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2377.652940] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2388.448878] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.462620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2388.467765] Lustre: Skipped 1 previous similar message [ 2392.122093] Lustre: server umount lustre-MDT0000 complete [ 2393.573097] LustreError: 69166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2393.602834] LustreError: 69166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2397.955494] LustreError: 64644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845567 with bad export cookie 17992899842915498584 [ 2397.958927] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2397.969605] LustreError: 64644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2398.746599] Lustre: server umount lustre-MDT0001 complete [ 2407.652424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2412.482971] Lustre: DEBUG MARKER: files compared the same [ 2414.879174] Lustre: 63572:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788845568/real 1788845568] req@00000000d7dd9cbe x1875740220030720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788845584 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2414.932669] Lustre: 63572:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 2414.971512] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.006780] Lustre: Skipped 1 previous similar message [ 2416.317173] Lustre: server umount lustre-OST0000 complete [ 2437.634355] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2440.713442] Key type lgssc unregistered [ 2440.978742] LNet: 74393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2440.986311] LNetError: 74393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2441.006767] LNet: Removed LNI 192.168.202.110@tcp [ 2441.964594] Key type .llcrypt unregistered [ 2441.967132] Key type ._llcrypt unregistered [ 2467.009589] Key type ._llcrypt registered [ 2467.011365] Key type .llcrypt registered [ 2467.126589] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2468.398412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2468.707600] alg: No test for adler32 (adler32-zlib) [ 2469.801921] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2470.124571] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2471.783364] Key type lgssc registered [ 2472.978415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2480.694706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2480.717338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.310417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2486.838478] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug +malloc all [ 2492.899261] LustreError: 75807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.935179] LustreError: 75807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2496.208752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.466352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2501.262549] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug +malloc all [ 2510.134955] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2516.159272] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2523.213616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2523.594788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.737387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2530.377731] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug +malloc all [ 2538.274593] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2552.288319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2552.291133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2552.305608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2552.321729] Lustre: Skipped 1 previous similar message [ 2552.336491] Lustre: Skipped 1 previous similar message [ 2556.954504] Lustre: server umount lustre-OST0000 complete [ 2562.530512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.532677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2562.544320] Lustre: Skipped 1 previous similar message [ 2562.560408] Lustre: Skipped 1 previous similar message [ 2567.651322] LustreError: 76807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.944762] Lustre: server umount lustre-MDT0000 complete [ 2572.768472] LustreError: 75800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.790962] LustreError: 75800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2572.929330] LustreError: 75786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845742 with bad export cookie 4602423126474247575 [ 2572.933759] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2572.939910] LustreError: 75786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2573.276651] Lustre: server umount lustre-MDT0001 complete [ 2585.916914] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2589.192802] Key type lgssc unregistered [ 2589.515881] LNet: 79352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2589.519216] LNetError: 79352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2589.539981] LNet: Removed LNI 192.168.202.110@tcp [ 2590.535889] Key type .llcrypt unregistered [ 2590.537484] Key type ._llcrypt unregistered [ 2615.818994] Key type ._llcrypt registered [ 2615.820989] Key type .llcrypt registered [ 2615.933996] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2617.490343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2617.578641] alg: No test for adler32 (adler32-zlib) [ 2618.691624] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2619.128351] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2620.895432] Key type lgssc registered [ 2622.168812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2631.493362] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2631.507810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2647.071414] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2652.192205] LustreError: 80739:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.110@tcp: failed processing log, type 4: rc = -110 [ 2678.879283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2686.329965] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2696.014402] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2705.343921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.474579] Lustre: *** cfs_fail_loc=706, val=0*** [ 2705.977126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2706.054299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2711.029434] LustreError: 80765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.044399] LustreError: 80765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2711.803649] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2714.082610] LustreError: 81824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.116038] LustreError: 81824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2719.205781] LustreError: 81823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2719.224531] LustreError: 81823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2722.032603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.854955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2728.544527] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2736.609020] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2743.132577] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2746.855288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2746.862730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.881650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2748.390572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.395500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2748.399523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2751.504824] Lustre: server umount lustre-OST0000 complete [ 2758.625247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.625984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2758.637913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2758.665732] Lustre: Skipped 2 previous similar messages [ 2763.746245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.759947] Lustre: Skipped 1 previous similar message [ 2768.867518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2768.877156] Lustre: Skipped 1 previous similar message [ 2769.375177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2769.584525] Lustre: server umount lustre-MDT0000 complete [ 2773.984396] LustreError: 81829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.016401] LustreError: 81829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2774.210049] LustreError: 80745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845944 with bad export cookie 5339366185057495194 [ 2774.214528] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2774.227556] LustreError: 80745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2774.508838] Lustre: server umount lustre-MDT0001 complete [ 2785.051759] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2787.638761] Key type lgssc unregistered [ 2787.945032] LNet: 84354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2787.953121] LNetError: 84354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2787.971807] LNet: Removed LNI 192.168.202.110@tcp [ 2788.843222] Key type .llcrypt unregistered [ 2788.845324] Key type ._llcrypt unregistered [ 2813.662721] Key type ._llcrypt registered [ 2813.668709] Key type .llcrypt registered [ 2813.777908] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2815.298222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2815.411166] alg: No test for adler32 (adler32-zlib) [ 2816.455785] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2816.723900] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2818.455237] Key type lgssc registered [ 2820.165871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2829.829707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2829.855024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.158548] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2846.872187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2847.646677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2852.827803] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2852.843715] LustreError: 86454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.859860] LustreError: 86454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2857.954971] LustreError: 86458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.975298] LustreError: 86458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2862.968279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2863.072309] LustreError: 86457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.095040] LustreError: 86457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2863.281578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2868.709052] LustreError: 86458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2868.746353] LustreError: 86458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2869.974886] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2872.809967] LustreError: 86457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2872.834804] LustreError: 86457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2878.973951] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2878.990122] Lustre: MGS: Client 47ae4273-a320-487e-8430-b8ffea0afacf (at 0@lo) reconnecting [ 2879.029288] LustreError: 86468:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2879.285421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.370219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2884.379876] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2899.938621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2899.952298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2899.962958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2899.980106] Lustre: Skipped 1 previous similar message [ 2900.001799] Lustre: Skipped 1 previous similar message [ 2904.356167] Lustre: server umount lustre-OST0000 complete [ 2910.178840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2910.206930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2910.207178] Lustre: Skipped 1 previous similar message [ 2910.208396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2910.231544] Lustre: Skipped 1 previous similar message [ 2914.555940] Lustre: server umount lustre-MDT0000 complete [ 2915.303522] LustreError: 86453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2915.319231] LustreError: 86453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2919.022235] Lustre: server umount lustre-MDT0001 complete [ 2923.416337] Lustre: server umount MGS complete [ 2937.222694] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:41:44 (1788846104) [ 2973.811432] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 2984.121812] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3022.454142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3027.591541] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3038.015952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3038.262680] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3038.299417] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3038.379295] Lustre: lustre-MDT0001: new disk, initializing [ 3038.473416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3038.510450] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3038.518912] Lustre: 93637:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3038.532487] LustreError: 93637:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3042.825094] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3052.652655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3052.880994] Lustre: lustre-OST0000: new disk, initializing [ 3052.884618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3052.889749] Lustre: 94403:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3052.944975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.895954] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3063.330929] Lustre: 94890:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a3b32efb x1875740968656384/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788846249 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3068.488642] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3075.415359] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3075.432757] Lustre: MGS: Client 95079630-df8e-4c88-9d7e-0deaee01a6f7 (at 0@lo) reconnecting [ 3075.502159] Lustre: 95448:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3075.525176] LustreError: 93623:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3075.540048] LustreError: 93623:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3075.561424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3075.703913] Lustre: lustre-MDT0000: new disk, initializing [ 3075.898639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.933220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3076.115082] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3077.026997] Lustre: 95472:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000099fbf8d9 x1875740968662784/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788846262 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3080.916995] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3083.743738] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3086.131142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3086.142448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3086.290402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3116.011839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.032213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3116.049427] Lustre: Skipped 1 previous similar message [ 3116.079137] Lustre: Skipped 1 previous similar message [ 3118.684713] Lustre: server umount lustre-OST0000 complete [ 3126.243278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3126.250626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3126.267513] Lustre: Skipped 1 previous similar message [ 3126.275386] Lustre: Skipped 1 previous similar message [ 3129.686258] Lustre: server umount lustre-MDT0000 complete [ 3131.362933] LustreError: 93609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3131.378971] LustreError: 93609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3140.022204] Lustre: server umount lustre-MDT0001 complete [ 3144.789733] Lustre: server umount MGS complete [ 3159.380939] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:45:27 (1788846327) [ 3178.836481] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3181.514972] Key type lgssc unregistered [ 3181.822937] LNet: 97972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3181.834757] LNetError: 97972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3181.856378] LNet: Removed LNI 192.168.202.110@tcp [ 3182.617422] Key type .llcrypt unregistered [ 3182.621529] Key type ._llcrypt unregistered [ 3203.604213] Key type ._llcrypt registered [ 3203.609066] Key type .llcrypt registered [ 3203.805441] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3204.693261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3204.783244] alg: No test for adler32 (adler32-zlib) [ 3206.052545] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 3206.243035] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3207.879298] Key type lgssc registered [ 3209.011783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3218.138721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3218.189727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.626336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.345677] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3230.178224] LustreError: 99189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.199617] LustreError: 99189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3233.150611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3233.769862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3237.828891] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3246.839108] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3254.036363] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3263.861578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3264.181129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.226774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3272.184450] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3279.656096] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3289.073251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3289.077624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.083932] Lustre: Skipped 1 previous similar message [ 3289.093521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3289.103100] Lustre: Skipped 1 previous similar message [ 3293.629366] Lustre: server umount lustre-OST0000 complete [ 3299.299072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.314836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3304.419645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3304.435943] Lustre: Skipped 2 previous similar messages [ 3304.976439] Lustre: server umount lustre-MDT0000 complete [ 3308.544929] LustreError: 99189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3308.571706] LustreError: 99189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3309.780944] LustreError: 99169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846479 with bad export cookie 3836219003560849105 [ 3309.806971] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3309.815253] LustreError: 99169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3310.704799] Lustre: server umount lustre-MDT0001 complete [ 3323.314482] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3326.898919] Key type lgssc unregistered [ 3327.114913] LNet: 102721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3327.118043] LNetError: 102721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3328.170893] LNet: Removed LNI 192.168.202.110@tcp [ 3329.344795] Key type .llcrypt unregistered [ 3329.349276] Key type ._llcrypt unregistered [ 3347.668448] Key type ._llcrypt registered [ 3347.672421] Key type .llcrypt registered [ 3347.797621] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3349.208031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3349.570376] alg: No test for adler32 (adler32-zlib) [ 3350.660784] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 3351.032136] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3352.775737] Key type lgssc registered [ 3353.794967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3360.907274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3360.907336] LustreError: 103819:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3360.911768] Lustre: Skipped 1 previous similar message [ 3360.924775] LustreError: 103818:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3360.930584] LustreError: 103819:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 3360.936533] LustreError: 103819:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3360.945352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3360.953995] LustreError: 103819:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3362.765480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.544709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.928336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3378.054747] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3383.814306] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing lsmod [ 3386.615748] LustreError: 105026:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3386.639523] LustreError: 105027:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3386.648405] LustreError: 105026:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4981 [ 3386.654582] LustreError: 105026:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3386.675706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3386.677854] LustreError: 105026:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3387.244533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.288022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3393.505909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3393.506943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.507892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3393.529026] Lustre: Skipped 2 previous similar messages [ 3397.675982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3397.695680] Lustre: Skipped 3 previous similar messages [ 3402.727922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3402.744080] Lustre: Skipped 3 previous similar messages [ 3404.255123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3404.476366] Lustre: server umount lustre-MDT0000 complete [ 3407.852276] LustreError: 103858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3407.871462] LustreError: 103858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3409.235643] LustreError: 103834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846579 with bad export cookie 1370646271726057320 [ 3409.259217] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3410.079568] LustreError: 104289:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bdccfae6 x1875741528579200/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3410.329134] Lustre: server umount lustre-MDT0001 complete [ 3425.322332] Lustre: server umount lustre-OST0000 complete [ 3436.401762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.902469] LustreError: 106090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.983537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3442.146511] LustreError: 106091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3442.274470] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3446.242685] LustreError: 106090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.370423] LustreError: 106091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.559054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.198318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3457.979412] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3466.576966] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3472.287747] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3478.945842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3479.335705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3481.407461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3486.062583] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3495.472991] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3512.288276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3512.298864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3512.318063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3513.317737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.475719] Lustre: server umount lustre-OST0000 complete [ 3519.971374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3519.982594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3520.005390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3520.022533] Lustre: Skipped 2 previous similar messages [ 3526.118895] Lustre: server umount lustre-MDT0000 complete [ 3528.676707] LustreError: 107095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3528.696808] LustreError: 107095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3530.056292] LustreError: 106070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846699 with bad export cookie 1370646271726058251 [ 3530.058697] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3530.072838] LustreError: 106070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3530.549624] Lustre: server umount lustre-MDT0001 complete [ 3541.939570] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3544.765664] Key type lgssc unregistered [ 3545.053110] LNet: 109645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3545.061586] LNetError: 109645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3545.090840] LNet: Removed LNI 192.168.202.110@tcp [ 3546.032601] Key type .llcrypt unregistered [ 3546.034915] Key type ._llcrypt unregistered [ 3571.759610] Key type ._llcrypt registered [ 3571.767165] Key type .llcrypt registered [ 3571.938655] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3573.570336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3573.627028] alg: No test for adler32 (adler32-zlib) [ 3574.685647] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 3574.923772] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3576.663175] Key type lgssc registered [ 3577.914820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3586.419832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3586.437985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3588.028722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.949466] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3597.291915] LustreError: 111058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.311932] LustreError: 111058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3601.870075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3602.382765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3607.452130] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3615.234319] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3622.139352] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3630.289543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3630.730143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3632.844437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3638.254251] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3645.738255] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3652.480347] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3657.885725] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3663.849768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.855217] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3663.861722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3663.880312] LustreError: Skipped 1 previous similar message [ 3668.011836] Lustre: server umount lustre-OST0000 complete [ 3674.083143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.085178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3674.088361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3674.088369] Lustre: Skipped 1 previous similar message [ 3674.106641] Lustre: Skipped 2 previous similar messages [ 3678.976589] Lustre: server umount lustre-MDT0000 complete [ 3679.203655] LustreError: 111058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3679.220935] LustreError: 111058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3683.297463] LustreError: 111054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3683.324949] LustreError: 111054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3683.409004] LustreError: 113592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846853 with bad export cookie 14783256708557968072 [ 3683.412098] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3683.416200] LustreError: 113592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3683.748924] Lustre: server umount lustre-MDT0001 complete [ 3698.267970] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3702.084195] Key type lgssc unregistered [ 3702.407089] LNet: 114700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3702.416983] LNetError: 114700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3703.462632] LNet: Removed LNI 192.168.202.110@tcp [ 3704.634451] Key type .llcrypt unregistered [ 3704.636705] Key type ._llcrypt unregistered [ 3726.528422] Key type ._llcrypt registered [ 3726.531854] Key type .llcrypt registered [ 3726.682288] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3729.448783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3729.474580] alg: No test for adler32 (adler32-zlib) [ 3730.671551] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 3730.911454] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3732.538687] Key type lgssc registered [ 3733.912359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3743.788546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3743.834458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3745.522534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3750.642031] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3756.012257] LustreError: 115920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3756.036985] LustreError: 115920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3760.152403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3760.622866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3765.244162] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3774.483179] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3782.030899] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3790.580590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3790.996676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3799.077323] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3808.077453] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3815.699843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3831.268777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3831.288880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.298062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3831.311658] Lustre: Skipped 1 previous similar message [ 3831.333607] Lustre: Skipped 1 previous similar message [ 3834.255817] Lustre: server umount lustre-OST0000 complete [ 3841.510338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3841.527218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3841.530127] Lustre: Skipped 1 previous similar message [ 3844.950065] Lustre: server umount lustre-MDT0000 complete [ 3846.630724] LustreError: 115921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3846.673124] LustreError: 115921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3849.325434] LustreError: 115901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847019 with bad export cookie 17096493389981038678 [ 3849.341229] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3849.344645] LustreError: 115901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3849.685424] Lustre: server umount lustre-MDT0001 complete [ 3862.477481] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3865.647397] Key type lgssc unregistered [ 3866.011974] LNet: 119474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3866.019265] LNetError: 119474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3866.034298] LNet: Removed LNI 192.168.202.110@tcp [ 3867.182485] Key type .llcrypt unregistered [ 3867.188092] Key type ._llcrypt unregistered [ 3896.484534] Key type ._llcrypt registered [ 3896.485814] Key type .llcrypt registered [ 3896.599240] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3897.701432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3897.774265] alg: No test for adler32 (adler32-zlib) [ 3898.966636] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 3899.288766] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3900.994337] Key type lgssc registered [ 3902.890553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3913.360734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3913.395518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.107062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3920.751448] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3924.524095] LustreError: 120908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3924.551680] LustreError: 120908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3929.582161] LustreError: 120909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3931.353637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3931.985398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3936.447383] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3945.866075] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3953.652897] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3961.546696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3962.041454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3969.391659] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3978.233650] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3984.648105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3995.812463] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4000.332642] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4000.345416] Lustre: Skipped 2 previous similar messages [ 4002.396777] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4002.445941] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4005.454268] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4005.457902] Lustre: Skipped 2 previous similar messages [ 4011.069905] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4011.074232] Lustre: Skipped 1 previous similar message [ 4019.430520] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.10@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4019.438741] Lustre: Skipped 2 previous similar messages [ 4020.116894] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.10@tcp 192.168.0.[2,10]@tcp [ 4027.803153] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.10/24@tcp in log lustre-MDT0000 [ 4027.809569] Lustre: Skipped 2 previous similar messages [ 4033.133681] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.10/24@tcp [ 4033.141182] Lustre: Skipped 1 previous similar message [ 4041.184767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4041.200619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4041.225146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4041.235632] Lustre: Skipped 1 previous similar message [ 4046.330456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4046.340148] Lustre: Skipped 1 previous similar message [ 4046.887846] Lustre: server umount lustre-OST0000 complete [ 4051.425929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.432463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4051.439856] Lustre: Skipped 2 previous similar messages [ 4051.459246] Lustre: Skipped 1 previous similar message [ 4056.544798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4056.555211] Lustre: Skipped 1 previous similar message [ 4061.665634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4061.683214] Lustre: Skipped 1 previous similar message [ 4065.247823] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4065.694534] Lustre: server umount lustre-MDT0000 complete [ 4066.785029] LustreError: 120904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4066.803269] LustreError: 120904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4070.198044] LustreError: 120889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847240 with bad export cookie 2757996915543865334 [ 4070.218051] LustreError: 120889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4070.220054] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4070.876245] Lustre: server umount lustre-MDT0001 complete [ 4082.687879] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4085.308854] Key type lgssc unregistered [ 4085.793417] LNet: 125785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4085.816622] LNetError: 125785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4086.889405] LNet: Removed LNI 192.168.202.110@tcp [ 4087.820145] Key type .llcrypt unregistered [ 4087.822288] Key type ._llcrypt unregistered [ 4121.414942] Key type ._llcrypt registered [ 4121.416587] Key type .llcrypt registered [ 4121.560788] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4122.752178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4122.784816] alg: No test for adler32 (adler32-zlib) [ 4124.264370] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 4124.484062] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 4126.191491] Key type lgssc registered [ 4127.454166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4135.850335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4135.879678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4137.375996] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4137.415706] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.10/24@tcp [ 4137.443125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4141.722381] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4147.684922] LustreError: 127359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.705420] LustreError: 127359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4151.189148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4151.594218] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4151.627818] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.10/24@tcp [ 4151.701374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4157.575680] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4166.487845] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4172.997754] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4180.568706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4180.964515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4183.073455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4188.242025] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4197.484429] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4211.527314] Lustre: server umount lustre-OST0000 complete [ 4216.801618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4216.809079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4216.816668] Lustre: Skipped 1 previous similar message [ 4221.931636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4221.942839] Lustre: Skipped 2 previous similar messages [ 4226.100519] Lustre: server umount lustre-MDT0000 complete [ 4227.043624] LustreError: 127360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.055106] LustreError: 127360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4228.065214] Lustre: 126669:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847381/real 1788847381] req@000000003b282ab7 x1875742340185216/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788847397 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4228.116334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.140251] LustreError: 128309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4228.165312] LustreError: 128309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4231.159173] LustreError: 129908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847400 with bad export cookie 13961532117968508045 [ 4231.166592] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4231.172858] LustreError: 129908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4231.495807] Lustre: server umount lustre-MDT0001 complete [ 4248.325583] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4252.161208] Key type lgssc unregistered [ 4252.439701] LNet: 130949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4252.448637] LNetError: 130949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4252.458301] LNet: Removed LNI 192.168.202.110@tcp [ 4253.386826] Key type .llcrypt unregistered [ 4253.389151] Key type ._llcrypt unregistered [ 4287.984490] Key type ._llcrypt registered [ 4287.987664] Key type .llcrypt registered [ 4288.131630] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 4303.823955] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4304.809260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4304.998742] alg: No test for adler32 (adler32-zlib) [ 4306.111312] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 4306.390749] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 4308.048269] Key type lgssc registered [ 4309.668957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4351.490689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4351.515587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4352.920670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4352.948258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4353.109890] Lustre: lustre-MDT0000: new disk, initializing [ 4353.250123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.287121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4359.108355] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4369.867323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4369.994399] Lustre: 134346:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4370.062665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4370.067463] Lustre: Skipped 1 previous similar message [ 4370.124153] Lustre: lustre-MDT0001: new disk, initializing [ 4370.197471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4370.236957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4370.260231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4374.293091] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4383.856819] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4391.731453] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4401.477466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4402.002936] Lustre: lustre-OST0000: new disk, initializing [ 4402.006391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4402.016425] Lustre: 136314:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4402.287053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.440973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4409.449249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4409.676752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4410.580743] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4423.624605] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4428.459387] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4428.755967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4433.214222] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4433.447393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4440.557398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4440.567226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.575625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4440.587904] Lustre: Skipped 1 previous similar message [ 4440.609412] Lustre: Skipped 1 previous similar message [ 4441.802605] Lustre: server umount lustre-OST0000 complete [ 4445.666984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.679475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4445.687440] Lustre: Skipped 1 previous similar message [ 4450.795509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4450.801693] Lustre: Skipped 1 previous similar message [ 4451.371613] Lustre: server umount lustre-MDT0000 complete [ 4455.830875] LustreError: 134339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847625 with bad export cookie 2047119362580942125 [ 4455.834633] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.839833] LustreError: 134339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4456.477294] Lustre: server umount lustre-MDT0001 complete [ 4467.946254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4468.408315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4472.956775] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4473.826885] LustreError: 138533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4473.854732] LustreError: 138533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4478.964214] LustreError: 138534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4483.787948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.066045] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4484.244497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4490.505922] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4500.295478] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4507.906535] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4517.035456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4517.327547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4524.948844] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4532.972135] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4538.169748] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4539.581589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4544.274313] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4544.492849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4553.269074] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4553.460875] Lustre: lustre-OST0001: new disk, initializing [ 4553.464301] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4553.469852] Lustre: 141912:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4553.565125] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4559.921961] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4561.046712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4561.071118] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4561.275752] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4568.101744] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4568.328695] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4573.143279] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4573.501234] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4580.535966] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e4814685800.ost_server_uuid 50 [ 4582.261188] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e4814685800.ost_server_uuid in IDLE state after 0 sec [ 4594.753866] Lustre: server umount lustre-OST0001 complete [ 4597.222538] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4597.231072] LustreError: Skipped 1 previous similar message [ 4597.241295] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4597.259623] LustreError: 140355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4597.260741] Lustre: Skipped 2 previous similar messages [ 4597.289412] LustreError: 140355:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4600.288631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4600.299896] LustreError: Skipped 1 previous similar message [ 4600.302942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4600.311664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4600.316048] Lustre: Skipped 1 previous similar message [ 4602.352696] LustreError: 140355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4602.368017] LustreError: 140355:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4605.391554] Lustre: server umount lustre-OST0000 complete [ 4612.067160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4612.079696] LustreError: Skipped 1 previous similar message [ 4612.086794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4612.106356] Lustre: Skipped 1 previous similar message [ 4612.111504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4612.121600] Lustre: Skipped 2 previous similar messages [ 4616.024602] Lustre: server umount lustre-MDT0000 complete [ 4617.702850] LustreError: 138528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4617.727696] LustreError: 138528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4619.404806] LustreError: 138514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847789 with bad export cookie 2047119362580943007 [ 4619.409936] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4619.411092] LustreError: 138514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4619.687803] Lustre: server umount lustre-MDT0001 complete [ 4646.164534] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4649.466464] Key type lgssc unregistered [ 4649.893362] LNet: 144917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4649.901493] LNetError: 144917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4649.916546] LNet: Removed LNI 192.168.202.110@tcp [ 4651.122139] Key type .llcrypt unregistered [ 4651.123437] Key type ._llcrypt unregistered [ 4693.920370] Key type ._llcrypt registered [ 4693.923250] Key type .llcrypt registered [ 4694.066555] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 4709.138686] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4710.595482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4710.724767] alg: No test for adler32 (adler32-zlib) [ 4711.736716] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 4711.913965] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 4713.551203] Key type lgssc registered [ 4714.717588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4749.783873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4749.820358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.186749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4751.214483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4751.282347] Lustre: lustre-MDT0000: new disk, initializing [ 4751.340270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4751.359885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4756.153689] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4767.685470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4767.924444] Lustre: 149066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4768.048716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4768.059871] Lustre: Skipped 1 previous similar message [ 4768.234448] Lustre: lustre-MDT0001: new disk, initializing [ 4768.412103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4768.464186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4768.478905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4773.719482] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4783.512822] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4789.685725] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4796.778977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4797.111917] Lustre: lustre-OST0000: new disk, initializing [ 4797.119475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4797.130865] Lustre: 151035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4797.243822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4803.751707] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4807.239180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4807.255822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4807.397981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4813.992690] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4819.293454] Lustre: Failing over lustre-OST0000 [ 4819.464050] Lustre: server umount lustre-OST0000 complete [ 4836.320046] Lustre: 147291:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847990/real 1788847990] req@00000000bdf42a18 x1875742955700736/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788848006 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4836.359355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4838.368421] Lustre: 147290:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847992/real 1788847992] req@000000004cf9f3b2 x1875742955701632/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788848008 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4838.393334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4839.059899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4839.445313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4839.483601] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4840.482506] Lustre: 147292:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847992/real 1788847992] req@0000000059f2c480 x1875742955701760/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788848008 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4843.492792] Lustre: 147292:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847997/real 1788847997] req@00000000d13ad51e x1875742955702784/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788848013 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4843.532207] Lustre: 147292:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4846.535920] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4846.563354] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4848.686811] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4848.697413] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4849.634640] Lustre: 147289:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788848003/real 1788848003] req@0000000001e02a0f x1875742955704064/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788848019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4849.668224] Lustre: 147289:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4850.754566] Lustre: Failing over lustre-MDT0000 [ 4851.017694] Lustre: server umount lustre-MDT0000 complete [ 4851.686153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4851.695389] Lustre: Skipped 1 previous similar message [ 4856.803217] LustreError: 149080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4856.824567] LustreError: 149080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4858.772516] LustreError: 149076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4861.928313] LustreError: 149074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4861.948769] LustreError: 149074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4867.043039] LustreError: 149074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4867.071302] LustreError: 149074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4868.065055] Lustre: 147289:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788848021/real 1788848021] req@0000000012ffbe67 x1875742955716480/t0(0) o400->MGC192.168.202.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788848037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4868.087255] Lustre: 147289:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4868.101922] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4870.759207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4873.190320] LustreError: 149079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4873.208784] LustreError: 149079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4877.285578] LustreError: 147288:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000032dbf8a3 x1875742955721984/t0(0) o250->MGC192.168.202.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4877.610761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4877.663344] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4879.256612] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4882.777899] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4882.947122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4882.957961] Lustre: Skipped 1 previous similar message [ 4882.986690] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4883.033810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4893.160523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.168278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4893.175895] Lustre: Skipped 2 previous similar messages [ 4893.190764] Lustre: Skipped 1 previous similar message [ 4898.284355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4898.290118] Lustre: Skipped 1 previous similar message [ 4898.546087] Lustre: server umount lustre-OST0000 complete [ 4903.405823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4903.408952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4903.418790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4908.514660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4908.523327] Lustre: Skipped 2 previous similar messages [ 4909.408175] Lustre: server umount lustre-MDT0000 complete [ 4912.921434] LustreError: 149863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848082 with bad export cookie 17492359386522739134 [ 4912.933423] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4913.272899] Lustre: server umount lustre-MDT0001 complete [ 4927.802396] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4931.721066] Key type lgssc unregistered [ 4932.396969] LNet: 154691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4932.410285] LNetError: 154691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4933.477312] LNet: Removed LNI 192.168.202.110@tcp [ 4934.600938] Key type .llcrypt unregistered [ 4934.612620] Key type ._llcrypt unregistered [ 4975.703425] Key type ._llcrypt registered [ 4975.705888] Key type .llcrypt registered [ 4975.797698] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4977.146756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4977.289823] alg: No test for adler32 (adler32-zlib) [ 4978.360808] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 4978.755193] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 4980.567116] Key type lgssc registered [ 4981.840158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4991.624769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4991.661932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4993.378368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.119868] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5003.745984] LustreError: 156210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.784751] LustreError: 156210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5008.876468] LustreError: 156211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5009.324306] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5013.720593] LustreError: 156239:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5013.727891] LustreError: 156239:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5013.733362] LustreError: 156239:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 5014.035946] Lustre: server umount lustre-MDT0000 complete [ 5029.150779] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 5040.489914] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5078.048449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5078.538703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5078.592461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5078.839680] Lustre: lustre-MDT0000: new disk, initializing [ 5079.022781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5079.051458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5083.837516] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5094.604666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5094.780546] Lustre: 159544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5094.896963] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5094.903976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5094.912694] Lustre: Skipped 1 previous similar message [ 5094.969420] Lustre: lustre-MDT0001: new disk, initializing [ 5095.054037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5095.108707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5095.121402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5099.898981] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5110.112158] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5116.092286] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5123.588409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5123.902420] Lustre: lustre-OST0000: new disk, initializing [ 5123.906683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5123.916317] Lustre: 161513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5124.023669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5130.665133] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5131.847029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5131.860852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5131.976916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5140.602756] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5145.329978] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5147.253517] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:18:35 (1788848315) [ 5397.480570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5397.488532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5397.506316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5398.499068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5398.513211] Lustre: Skipped 1 previous similar message [ 5398.523554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5398.526236] Lustre: Skipped 2 previous similar messages [ 5403.617979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5403.629239] Lustre: Skipped 2 previous similar messages [ 5403.915669] Lustre: server umount lustre-MDT0000 complete [ 5407.953197] LustreError: 159535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848577 with bad export cookie 14698599456863592800 [ 5407.962320] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5407.985835] LustreError: 159535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5408.758678] LustreError: 159549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5408.759050] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5408.808455] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5412.767994] LustreError: 159549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5414.526767] Lustre: server umount lustre-MDT0001 complete [ 5425.620454] Lustre: server umount lustre-OST0000 complete [ 5437.393543] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 5447.153068] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5488.593520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5488.880738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5488.908908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5488.977955] Lustre: lustre-MDT0000: new disk, initializing [ 5489.127666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5489.143631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5493.972703] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5505.165746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5505.260702] Lustre: 166176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5505.328782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5505.332590] Lustre: Skipped 1 previous similar message [ 5505.419097] Lustre: lustre-MDT0001: new disk, initializing [ 5505.514433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5505.549619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5505.567123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5510.496980] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5521.525258] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5529.340762] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5537.091357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5537.444445] Lustre: lustre-OST0000: new disk, initializing [ 5537.447871] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5537.453038] Lustre: 168146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5537.596659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5544.798248] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5546.046896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5546.061615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5546.188539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5555.084800] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5560.672222] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5560.901243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5566.642205] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5566.983857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5572.066556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5572.072146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5572.086358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5572.086367] Lustre: Skipped 1 previous similar message [ 5575.527378] Lustre: server umount lustre-OST0000 complete [ 5582.308622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.309914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5582.340060] Lustre: Skipped 1 previous similar message [ 5591.520541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5591.528973] Lustre: Skipped 6 previous similar messages [ 5594.080736] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5596.344397] Lustre: server umount lustre-MDT0000 complete [ 5596.640887] LustreError: 166189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5596.657080] LustreError: 166189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5600.122188] LustreError: 169687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848769 with bad export cookie 14698599456863751637 [ 5600.127375] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5600.130041] LustreError: 169687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5600.489144] Lustre: server umount lustre-MDT0001 complete [ 5610.552319] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:26:17 (1788848777) [ 5628.815801] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 5640.094925] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5681.178310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5681.436123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5681.487597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5681.563258] Lustre: lustre-MDT0000: new disk, initializing [ 5681.714996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.736594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5686.881235] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5699.632552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.731410] Lustre: 172905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5700.091790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5705.956492] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5717.581430] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5723.171613] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5730.336774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5730.686689] Lustre: lustre-OST0000: new disk, initializing [ 5730.692202] Lustre: Skipped 1 previous similar message [ 5730.701933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5730.703846] Lustre: Skipped 2 previous similar messages [ 5730.719946] Lustre: 174883:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5737.351687] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5737.531822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5737.544504] Lustre: Skipped 1 previous similar message [ 5737.551763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5737.681757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5745.648428] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5753.325502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5753.335410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.352965] Lustre: Skipped 1 previous similar message [ 5753.358894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5753.366248] Lustre: Skipped 1 previous similar message [ 5758.340260] Lustre: server umount lustre-OST0000 complete [ 5776.351404] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5776.537986] Lustre: server umount lustre-MDT0000 complete [ 5778.927913] LustreError: 172915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.948313] LustreError: 172915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5780.677364] LustreError: 172897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848950 with bad export cookie 14698599456863752519 [ 5780.686707] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5780.691375] LustreError: 172897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5781.141128] Lustre: server umount lustre-MDT0001 complete [ 5789.889338] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:29:17 (1788848957) [ 5806.744838] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 5818.548324] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5850.196626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5850.450756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5850.487841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5850.572892] Lustre: lustre-MDT0000: new disk, initializing [ 5850.645619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5850.648783] Lustre: Skipped 2 previous similar messages [ 5850.660137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5854.911533] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5864.969360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.077295] Lustre: 179204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5865.353423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5869.826554] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5879.357259] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5884.918536] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5891.289351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5891.526794] Lustre: 181183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5897.280664] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5898.826470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5898.970070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5905.974742] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5914.593949] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5914.614170] LustreError: Skipped 1 previous similar message [ 5914.614837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5914.623154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5914.623166] Lustre: Skipped 6 previous similar messages [ 5914.658666] Lustre: Skipped 4 previous similar messages [ 5918.464794] Lustre: server umount lustre-OST0000 complete [ 5928.376419] Lustre: server umount lustre-MDT0000 complete [ 5929.952954] LustreError: 179218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.981239] LustreError: 179218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5931.994197] LustreError: 179197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849101 with bad export cookie 14698599456863753604 [ 5932.001337] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5932.010506] LustreError: 179197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5932.393173] Lustre: server umount lustre-MDT0001 complete [ 5943.938769] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 5946.905609] Key type lgssc unregistered [ 5947.190677] LNet: 183038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5947.199635] LNetError: 183038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5947.224533] LNet: Removed LNI 192.168.202.110@tcp [ 5948.030690] Key type .llcrypt unregistered [ 5948.034815] Key type ._llcrypt unregistered [ 5973.884950] Key type ._llcrypt registered [ 5973.888304] Key type .llcrypt registered [ 5974.014605] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5974.982477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5975.173366] alg: No test for adler32 (adler32-zlib) [ 5976.428247] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 5976.693095] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 5978.386639] Key type lgssc registered [ 5979.779092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5987.990154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5988.029515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5989.642089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5994.817962] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6000.102284] LustreError: 184450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6000.131837] LustreError: 184450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6004.438628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6004.772420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6009.234438] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6017.095306] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6023.168193] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6030.339238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6030.676539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6034.798699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6036.605782] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6045.110971] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6070.758110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6070.759467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.760279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6070.790717] Lustre: Skipped 1 previous similar message [ 6072.792939] Lustre: server umount lustre-OST0000 complete [ 6080.993455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6081.019245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6081.020716] Lustre: Skipped 1 previous similar message [ 6081.040251] Lustre: Skipped 2 previous similar messages [ 6083.072885] Lustre: server umount lustre-MDT0000 complete [ 6086.122650] LustreError: 184450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.141520] LustreError: 184450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6086.630934] LustreError: 184432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849256 with bad export cookie 6175832496651113542 [ 6086.646253] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6086.651078] LustreError: 184432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6086.940817] Lustre: server umount lustre-MDT0001 complete [ 6097.703456] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 6100.377591] Key type lgssc unregistered [ 6100.658701] LNet: 188013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6100.665406] LNetError: 188013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6101.734373] LNet: Removed LNI 192.168.202.110@tcp [ 6102.621429] Key type .llcrypt unregistered [ 6102.630262] Key type ._llcrypt unregistered [ 6124.640605] Key type ._llcrypt registered [ 6124.642519] Key type .llcrypt registered [ 6124.715496] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 6125.734673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6125.923841] alg: No test for adler32 (adler32-zlib) [ 6127.144646] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 6127.359638] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 6129.007189] Key type lgssc registered [ 6130.280949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6137.229419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6137.249048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6138.800827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6143.330976] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6149.096652] LustreError: 189428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6149.124317] LustreError: 189428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6151.498607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6151.966361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6156.146535] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6163.552616] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6169.071731] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6175.666523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6175.882053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.119383] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6184.113953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6186.969639] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6192.609776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6192.618112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6192.629708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6194.090770] Lustre: lustre-OST0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 6194.662189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6194.667259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6198.298972] Lustre: server umount lustre-OST0000 complete [ 6204.383907] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9e4809a3d800.ost_server_uuid 50 [ 6206.070862] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6220.135880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6220.141919] Lustre: Skipped 1 previous similar message [ 6220.142361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6220.158376] Lustre: Skipped 3 previous similar messages [ 6225.379126] LustreError: 189428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6225.396369] LustreError: 189428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6225.497391] Lustre: server umount lustre-MDT0000 complete [ 6228.661504] LustreError: 192006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849398 with bad export cookie 9731875596657866879 [ 6228.664497] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6228.669016] LustreError: 192006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6228.981345] Lustre: server umount lustre-MDT0001 complete [ 6236.045461] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:36:44 (1788849404) [ 6241.370687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6241.659804] LustreError: 193341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6241.735359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6245.619903] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6246.882572] LustreError: 193342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6252.001385] LustreError: 193341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6253.945408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.273259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6258.361349] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6266.353831] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6272.050688] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6279.200630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6279.458569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6286.455055] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6286.635548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6294.082749] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6300.845738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6301.107841] Lustre: lustre-OST0001: new disk, initializing [ 6301.112590] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6301.125296] Lustre: 196182:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6301.255345] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6307.714844] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6307.996078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6308.009429] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6308.166363] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6317.542570] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6321.648110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6321.657081] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6321.667170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6326.631200] Lustre: server umount lustre-OST0000 complete [ 6328.212120] LustreError: 196553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.701776] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6332.919089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6336.902907] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6337.078253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6349.285995] LustreError: 195161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6349.301931] LustreError: 195161:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6354.412249] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6354.420922] LustreError: Skipped 1 previous similar message [ 6354.424689] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6354.435836] Lustre: Skipped 1 previous similar message [ 6354.443527] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6354.450430] Lustre: Skipped 5 previous similar messages [ 6356.690682] Lustre: server umount lustre-OST0001 complete [ 6362.081021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6362.086083] LustreError: Skipped 1 previous similar message [ 6364.643062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6364.644903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.646448] Lustre: Skipped 1 previous similar message [ 6364.657161] Lustre: Skipped 2 previous similar messages [ 6366.540300] Lustre: server umount lustre-MDT0000 complete [ 6369.636909] LustreError: 196181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849539 with bad export cookie 9731875596657867950 [ 6369.648651] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6369.649836] LustreError: 196181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6369.906700] Lustre: server umount lustre-MDT0001 complete [ 6388.994734] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:39:17 (1788849557) [ 6394.419224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6394.611757] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6394.637971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6394.810138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6397.920829] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6405.372263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.460708] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6405.519140] Lustre: 199622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6409.674752] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6416.609032] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6421.719231] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6428.065071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6428.198915] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6434.767915] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6437.507686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6441.575811] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6448.123764] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6448.170435] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6454.378744] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6461.062583] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6466.029934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6466.038040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6466.056124] Lustre: Skipped 1 previous similar message [ 6466.079270] Lustre: Skipped 2 previous similar messages [ 6469.774698] Lustre: server umount lustre-OST0000 complete [ 6471.162421] LustreError: 201470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6471.190908] LustreError: 201470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6486.503993] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6486.511537] LustreError: Skipped 1 previous similar message [ 6490.540746] Lustre: server umount lustre-OST0001 complete [ 6499.863155] Lustre: server umount lustre-MDT0000 complete [ 6502.804900] LustreError: 199613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849672 with bad export cookie 9731875596657869357 [ 6502.810811] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6502.818981] LustreError: 199613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6503.058264] Lustre: server umount lustre-MDT0001 complete [ 6524.128237] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:41:32 (1788849692) [ 6539.173363] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 6548.375950] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 6581.658644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6581.805896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6581.827288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6581.890783] Lustre: lustre-MDT0000: new disk, initializing [ 6581.950069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6581.953371] Lustre: Skipped 3 previous similar messages [ 6581.964974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6586.003171] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6595.445757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6595.532815] Lustre: 207715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6595.571683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6595.574817] Lustre: Skipped 1 previous similar message [ 6595.627682] Lustre: lustre-MDT0001: new disk, initializing [ 6595.716494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6595.727489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6599.403900] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6608.043743] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6613.686338] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6619.453732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6619.719683] Lustre: lustre-OST0000: new disk, initializing [ 6619.724767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6619.732503] Lustre: 209684:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6624.610660] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6628.897835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6628.909902] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6629.046137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6632.931793] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6636.993722] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6637.198082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6641.177066] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6641.415563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6644.708282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6644.709333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6644.717879] LustreError: Skipped 1 previous similar message [ 6644.725822] Lustre: Skipped 4 previous similar messages [ 6644.726860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6644.726864] Lustre: Skipped 5 previous similar messages [ 6649.159682] Lustre: server umount lustre-OST0000 complete [ 6658.413893] Lustre: server umount lustre-MDT0000 complete [ 6660.065467] LustreError: 207727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6660.090823] LustreError: 207727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6661.446456] LustreError: 207705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849831 with bad export cookie 9731875596657870918 [ 6661.452118] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6661.464241] LustreError: 207705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6661.800557] Lustre: server umount lustre-MDT0001 complete [ 6668.307309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6672.896606] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6679.837969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6684.389666] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6691.884229] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6697.231931] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6704.235514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6710.942739] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6718.428472] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6722.690894] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6722.897280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6727.334195] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6727.554916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6731.233720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6731.239321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6731.248325] Lustre: Skipped 3 previous similar messages [ 6731.256171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6731.262153] Lustre: Skipped 3 previous similar messages [ 6735.712815] Lustre: server umount lustre-OST0000 complete [ 6741.197162] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6745.575648] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6772.873423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6773.116987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6773.119533] Lustre: Skipped 5 previous similar messages [ 6775.148994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6775.187053] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6778.407201] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6784.420556] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6784.443711] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6785.335624] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6793.086800] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 6799.848973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6799.855072] LustreError: Skipped 1 previous similar message [ 6802.521883] Lustre: server umount lustre-OST0000 complete [ 6812.760372] Lustre: server umount lustre-MDT0000 complete [ 6815.202561] LustreError: 211902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6815.223583] LustreError: 211902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6816.461772] LustreError: 211881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849986 with bad export cookie 9731875596657871800 [ 6816.462767] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6816.488728] LustreError: 211881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6816.953690] Lustre: server umount lustre-MDT0001 complete [ 6824.841810] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:46:32 (1788849992) [ 6830.851681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6835.906315] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6844.454821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6849.185477] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6857.076216] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6861.684361] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6867.052876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6873.007934] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6879.720812] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6884.306748] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6884.501771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6888.772978] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6888.968756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6894.819023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6894.981624] Lustre: lustre-OST0001: new disk, initializing [ 6894.984749] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6894.988695] Lustre: 221236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6900.852604] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6902.423030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6902.428821] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6902.490689] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6909.847132] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6913.612411] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6913.766150] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6918.520269] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6918.780748] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6923.235430] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6923.242479] Lustre: Skipped 5 previous similar messages [ 6923.246231] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6923.254694] Lustre: Skipped 8 previous similar messages [ 6926.801546] Lustre: server umount lustre-OST0001 complete [ 6932.172451] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6932.399467] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6936.295967] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6936.487904] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6964.182768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6969.858463] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6969.870825] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6969.877257] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6970.212809] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6976.765905] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6978.194189] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6980.066638] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6980.082944] LustreError: Skipped 3 previous similar messages [ 6994.399322] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6994.479766] Lustre: server umount lustre-OST0001 complete [ 7016.150783] Lustre: server umount lustre-OST0000 complete [ 7025.905414] Lustre: server umount lustre-MDT0000 complete [ 7029.058336] LustreError: 217848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850198 with bad export cookie 9731875596657872766 [ 7029.059471] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7029.069243] LustreError: 217848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7035.481359] Lustre: server umount lustre-MDT0001 complete [ 7055.268462] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:50:23 (1788850223) [ 7060.060945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.336304] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7060.372339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7060.499144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7060.506421] Lustre: Skipped 5 previous similar messages [ 7063.511496] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7068.654167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7068.755123] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7068.782758] Lustre: 226480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7072.145523] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7078.319465] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7082.631938] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7088.173207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7088.334108] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7093.289995] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7099.872123] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7108.005739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7108.048888] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7113.116247] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7119.193931] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7120.362437] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7124.301976] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7124.534602] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7128.798778] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7128.961301] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7133.435733] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e4808214800.ost_server_uuid 50 [ 7134.817314] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e4808214800.ost_server_uuid in FULL state after 0 sec [ 7135.808839] Lustre: Permanently deactivating lustre-OST0001 [ 7135.817468] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7138.518636] Lustre: Permanently reactivating lustre-OST0001 [ 7147.824310] Lustre: server umount lustre-OST0001 complete [ 7152.106989] LustreError: 230953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7152.133441] LustreError: 230953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 7157.008968] Lustre: server umount lustre-OST0000 complete [ 7168.322173] Lustre: server umount lustre-MDT0000 complete [ 7171.307774] LustreError: 226472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850341 with bad export cookie 9731875596657874054 [ 7171.310329] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7171.331733] LustreError: 226472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7171.543769] Lustre: server umount lustre-MDT0001 complete [ 7189.366328] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:52:37 (1788850357) [ 7195.078126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7195.371619] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7195.396157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7195.405924] Lustre: Skipped 5 previous similar messages [ 7199.035596] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7205.370347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7205.495367] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7205.560895] Lustre: 233170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7209.047834] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7214.972301] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7219.627439] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7224.520506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7224.641966] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7224.658212] Lustre: Permanently deactivating lustre-OST0000 [ 7224.664333] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7229.229791] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7232.014982] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7235.160418] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7239.850525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7239.917710] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7244.955537] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7247.527560] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7250.376919] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7251.663373] Lustre: Permanently reactivating lustre-OST0000 [ 7251.681644] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7251.687247] Lustre: Skipped 2 previous similar messages [ 7253.292289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7253.301155] Lustre: Skipped 12 previous similar messages [ 7253.304575] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7253.311154] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7253.319567] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7253.325908] Lustre: Skipped 1 previous similar message [ 7253.345511] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7253.357737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7263.200389] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7263.212264] LustreError: Skipped 6 previous similar messages [ 7263.217131] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7263.234597] Lustre: Skipped 24 previous similar messages [ 7264.970622] Lustre: server umount lustre-OST0001 complete [ 7282.143295] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7282.245618] Lustre: server umount lustre-OST0000 complete [ 7291.023663] Lustre: server umount lustre-MDT0000 complete [ 7293.225996] LustreError: 233160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850463 with bad export cookie 9731875596657875685 [ 7293.227308] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7293.232570] LustreError: 233160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7293.438910] Lustre: server umount lustre-MDT0001 complete [ 7301.220891] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 7303.275276] Key type lgssc unregistered [ 7303.597559] LNet: 238023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7303.603946] LNetError: 238023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7304.680640] LNet: Removed LNI 192.168.202.110@tcp [ 7305.316188] Key type .llcrypt unregistered [ 7305.318765] Key type ._llcrypt unregistered [ 7322.135245] Key type ._llcrypt registered [ 7322.137350] Key type .llcrypt registered [ 7322.220548] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 7322.950283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7322.960239] alg: No test for adler32 (adler32-zlib) [ 7323.978733] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 7324.209881] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 7325.831724] Key type lgssc registered [ 7326.603755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7334.095788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7334.110394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7335.464977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7338.191613] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7344.287884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7344.396620] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7344.427467] Lustre: 239454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7344.437289] Lustre: Permanently deactivating lustre-MDT0001 [ 7344.451693] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7344.767222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7348.009303] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7348.629074] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7353.022356] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7356.909485] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7362.008370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7362.254586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7366.783831] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7369.455266] LustreError: 241307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7369.470815] LustreError: 241307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7369.483880] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7372.111064] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7374.832172] LustreError: 241307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7377.144530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7377.285608] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7380.394531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7380.404922] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7380.420902] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7381.871691] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7387.182989] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7389.396777] Lustre: Permanently reactivating lustre-MDT0001 [ 7389.411725] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7389.416404] Lustre: Skipped 2 previous similar messages [ 7394.735023] Lustre: lustre-MDT0001: Client 4852ed06-352b-4b21-96ea-a371038945a7 (at 192.168.202.10@tcp) reconnecting [ 7395.876035] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7395.883748] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7395.890448] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7395.895899] LustreError: 243116:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7395.896786] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7395.899986] LustreError: 243116:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7395.906757] LustreError: 243116:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.110@tcp: cfg command failed: rc = -108 [ 7395.910135] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7395.910135] [ 7395.914245] LustreError: 239458:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7397.567295] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7398.501168] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7402.083914] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7402.246691] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7404.941757] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7405.094382] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7406.638747] Lustre: Permanently deactivating lustre-MDT0001 [ 7406.650443] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7406.656711] Lustre: Skipped 2 previous similar messages [ 7412.288687] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7421.408333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7421.412863] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7421.413802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7421.431957] Lustre: Skipped 3 previous similar messages [ 7424.574526] Lustre: server umount lustre-MDT0000 complete [ 7426.530967] LustreError: 239465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7426.541240] LustreError: 239465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7427.149268] LustreError: 239446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850596 with bad export cookie 10025996150529661264 [ 7427.153092] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7427.154288] LustreError: 239446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7427.401942] Lustre: server umount lustre-MDT0001 complete [ 7436.046834] Lustre: server umount lustre-OST0000 complete [ 7444.932797] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:56:53 (1788850613) [ 7453.191113] Lustre: server umount lustre-OST0001 complete [ 7458.534890] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 7465.258103] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 7495.302315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7495.858812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7495.887595] Lustre: Skipped 2 previous similar messages [ 7495.925541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7495.994193] Lustre: lustre-MDT0000: new disk, initializing [ 7496.169616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7496.209282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7504.272685] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7519.275430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.379767] Lustre: 247718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7519.433499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7519.438204] Lustre: Skipped 1 previous similar message [ 7519.529470] Lustre: lustre-MDT0001: new disk, initializing [ 7519.694352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7519.766098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7519.809366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7527.595469] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7541.346486] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7549.012978] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7557.517667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7557.876465] Lustre: lustre-OST0000: new disk, initializing [ 7557.881160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7557.888197] Lustre: 249689:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7557.982791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7563.392421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7563.408161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7563.548212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7564.550626] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7575.819568] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7578.144541] LustreError: 249697:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7587.795368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7587.974318] Lustre: lustre-OST0001: new disk, initializing [ 7587.977483] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7587.985122] Lustre: 250802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7588.193748] LustreError: 249697:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7588.210322] LustreError: 250613:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7594.312323] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7597.157313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7597.167467] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7597.298383] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7598.285702] LustreError: 250613:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7603.742461] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7605.339507] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7607.786304] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7607.790195] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7607.790946] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7607.797621] Lustre: Skipped 1 previous similar message [ 7607.819453] Lustre: Skipped 4 previous similar messages [ 7610.290248] Lustre: lustre-OST0001: Not available for connect from 192.168.202.10@tcp (stopping) [ 7612.912662] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7612.916339] Lustre: Skipped 1 previous similar message [ 7613.481961] Lustre: server umount lustre-OST0001 complete [ 7615.381058] LustreError: 249672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7623.141621] LustreError: 249670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7623.175685] LustreError: 249670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7630.304651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7630.309369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7630.317308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7630.320801] Lustre: Skipped 1 previous similar message [ 7633.400039] LustreError: 249672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7633.423066] LustreError: 249672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7636.561311] Lustre: server umount lustre-OST0000 complete [ 7642.592716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7642.597057] LustreError: Skipped 1 previous similar message [ 7642.598774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7642.619027] Lustre: Skipped 1 previous similar message [ 7642.629724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7642.637191] Lustre: Skipped 2 previous similar messages [ 7647.056988] Lustre: server umount lustre-MDT0000 complete [ 7650.732088] LustreError: 247710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850820 with bad export cookie 10025996150529664085 [ 7650.732088] LustreError: 247709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850820 with bad export cookie 10025996150529664085 [ 7650.742541] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7650.746647] LustreError: 247710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7665.119559] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7665.452179] Lustre: server umount lustre-MDT0001 complete [ 7677.922302] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 7681.121343] Key type lgssc unregistered [ 7681.380835] LNet: 252838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7681.386530] LNetError: 252838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7681.397281] LNet: Removed LNI 192.168.202.110@tcp [ 7682.425461] Key type .llcrypt unregistered [ 7682.431869] Key type ._llcrypt unregistered [ 7724.481604] Key type ._llcrypt registered [ 7724.484355] Key type .llcrypt registered [ 7724.614933] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 7726.590358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7726.895531] alg: No test for adler32 (adler32-zlib) [ 7728.452353] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 7729.033996] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 7730.831677] Key type lgssc registered [ 7732.603428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7742.262806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7742.278303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7743.483424] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7743.518900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7743.750698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7748.829667] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7758.007417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7758.189209] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7758.229474] Lustre: 255054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7758.475750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7763.622541] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7772.336325] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7778.911856] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7786.297446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7786.556321] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7786.795849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7794.285439] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7795.442629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7802.757198] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7857.120983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7857.136131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7857.145974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7857.180536] Lustre: Skipped 1 previous similar message [ 7862.149947] Lustre: server umount lustre-OST0000 complete [ 7866.442846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7879.662407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7879.935653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7886.917576] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7896.406400] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7897.277361] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7897.312448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7897.314395] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7897.366093] Lustre: Skipped 1 previous similar message [ 7938.529117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7938.533973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7938.546624] LustreError: Skipped 1 previous similar message [ 7938.573903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7938.581299] Lustre: Skipped 2 previous similar messages [ 7943.114247] Lustre: server umount lustre-OST0000 complete [ 7948.256768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7948.262467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7948.286374] Lustre: Skipped 1 previous similar message [ 7948.293606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7953.349641] Lustre: server umount lustre-MDT0000 complete [ 7956.846858] LustreError: 255044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851126 with bad export cookie 12270240888733393288 [ 7956.847713] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7956.858199] LustreError: 255044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7957.480247] Lustre: server umount lustre-MDT0001 complete [ 7972.596433] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 7976.381293] Key type lgssc unregistered [ 7976.639608] LNet: 260479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7976.643945] LNetError: 260479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7976.663776] LNet: Removed LNI 192.168.202.110@tcp [ 7977.874830] Key type .llcrypt unregistered [ 7977.878096] Key type ._llcrypt unregistered [ 8007.418246] Key type ._llcrypt registered [ 8007.421870] Key type .llcrypt registered [ 8007.616033] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8009.336217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8009.472057] alg: No test for adler32 (adler32-zlib) [ 8010.614199] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 8010.994658] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8012.759728] Key type lgssc registered [ 8014.201929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8023.933287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8023.967696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8025.522698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8030.936327] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8035.815619] LustreError: 261899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8035.839076] LustreError: 261899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8040.936170] LustreError: 261898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8040.952347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8041.298273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8046.155445] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8054.514706] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8062.194244] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8069.290292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8069.789759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8070.843571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8077.165972] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8085.960755] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8112.106763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8112.111345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8112.115314] LustreError: Skipped 1 previous similar message [ 8112.116426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8112.143454] Lustre: Skipped 1 previous similar message [ 8112.159420] Lustre: Skipped 1 previous similar message [ 8116.209890] Lustre: server umount lustre-OST0000 complete [ 8122.342720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8122.347633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8122.372696] Lustre: Skipped 1 previous similar message [ 8122.388420] Lustre: Skipped 1 previous similar message [ 8127.275235] Lustre: server umount lustre-MDT0000 complete [ 8127.460848] LustreError: 261898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8127.489868] LustreError: 261898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8131.650239] LustreError: 261880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851301 with bad export cookie 16229131003597781705 [ 8131.663164] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8131.665352] LustreError: 261880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8132.147785] Lustre: server umount lustre-MDT0001 complete [ 8145.037593] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8148.551528] Key type lgssc unregistered [ 8148.915269] LNet: 266206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8148.923051] LNetError: 266206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8149.989822] LNet: Removed LNI 192.168.202.110@tcp [ 8151.172900] Key type .llcrypt unregistered [ 8151.179777] Key type ._llcrypt unregistered [ 8170.955234] Key type ._llcrypt registered [ 8170.957413] Key type .llcrypt registered [ 8171.046695] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8172.039725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8172.079951] alg: No test for adler32 (adler32-zlib) [ 8173.246903] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 8173.494522] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8175.143329] Key type lgssc registered [ 8176.110471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8183.567427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8183.583288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8184.909598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8189.254105] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8195.047322] LustreError: 267428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8195.062267] LustreError: 267428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8198.901622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8199.372401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8204.153368] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8210.953788] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8217.335779] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8224.916609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8225.510272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8229.618600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8233.324452] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8240.845207] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8260.576504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8260.587867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8260.594346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8260.600461] Lustre: Skipped 1 previous similar message [ 8260.621367] Lustre: Skipped 1 previous similar message [ 8262.191373] Lustre: server umount lustre-OST0000 complete [ 8270.818057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8270.834458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8270.853032] Lustre: Skipped 1 previous similar message [ 8272.692885] Lustre: server umount lustre-MDT0000 complete [ 8275.946207] LustreError: 267427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8275.990299] LustreError: 267427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8277.178089] LustreError: 269270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851446 with bad export cookie 6970103684093480103 [ 8277.188574] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8277.193121] LustreError: 269270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8277.638571] Lustre: server umount lustre-MDT0001 complete [ 8290.233861] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8292.785140] Key type lgssc unregistered [ 8293.200596] LNet: 271106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8293.207891] LNetError: 271106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8294.245558] LNet: Removed LNI 192.168.202.110@tcp [ 8295.260899] Key type .llcrypt unregistered [ 8295.263243] Key type ._llcrypt unregistered [ 8322.976965] Key type ._llcrypt registered [ 8322.980579] Key type .llcrypt registered [ 8323.147857] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8324.529392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8324.786822] alg: No test for adler32 (adler32-zlib) [ 8326.055462] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 8326.532951] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8328.319364] Key type lgssc registered [ 8329.962402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8339.215198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8339.240091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8341.215667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8345.632284] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8350.691570] LustreError: 272521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8350.711522] LustreError: 272521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8355.820414] LustreError: 272520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8355.858746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8356.474294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8362.143993] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8371.104944] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8378.583929] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8386.335296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8386.589205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8393.552791] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8405.635832] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8411.200389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8436.706753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8436.714942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8436.738841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8441.831302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8441.850329] Lustre: Skipped 2 previous similar messages [ 8442.777856] Lustre: server umount lustre-OST0000 complete [ 8448.483522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8448.502787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8448.529230] Lustre: Skipped 1 previous similar message [ 8448.538367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8452.066701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8452.070355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8453.908214] Lustre: server umount lustre-MDT0000 complete [ 8457.184607] LustreError: 272521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8457.195905] LustreError: 272521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8459.429319] LustreError: 272500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851629 with bad export cookie 9901007363638974574 [ 8459.462702] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8459.464079] LustreError: 272500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8460.064046] Lustre: server umount lustre-MDT0001 complete [ 8476.133872] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8479.093361] Key type lgssc unregistered [ 8479.530120] LNet: 276849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8479.545585] LNetError: 276849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8479.564232] LNet: Removed LNI 192.168.202.110@tcp [ 8480.586067] Key type .llcrypt unregistered [ 8480.589203] Key type ._llcrypt unregistered [ 8502.552349] Key type ._llcrypt registered [ 8502.553732] Key type .llcrypt registered [ 8502.718390] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8503.964880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8504.004162] alg: No test for adler32 (adler32-zlib) [ 8505.247660] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 8505.535025] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8507.311190] Key type lgssc registered [ 8509.427847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8520.918262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8520.982301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8523.123534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8527.303844] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8532.448815] LustreError: 278082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8532.459041] LustreError: 278082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8537.347114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8537.727221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8542.848653] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8552.619088] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8559.648821] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8567.863648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8568.292398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8576.287215] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8586.276363] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8592.895483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8613.366034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8613.370039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8613.378638] LustreError: Skipped 1 previous similar message [ 8613.383880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8613.392802] Lustre: Skipped 1 previous similar message [ 8613.418320] Lustre: Skipped 1 previous similar message [ 8615.475761] Lustre: server umount lustre-OST0000 complete [ 8623.591978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8623.598971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8623.599368] Lustre: Skipped 1 previous similar message [ 8626.018120] Lustre: server umount lustre-MDT0000 complete [ 8628.705650] LustreError: 278095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8628.723348] LustreError: 278095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8629.863716] LustreError: 278047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851799 with bad export cookie 13501849985940962658 [ 8629.864901] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8629.871236] LustreError: 278047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8630.490328] Lustre: server umount lustre-MDT0001 complete [ 8641.295735] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8644.175841] Key type lgssc unregistered [ 8644.544424] LNet: 281808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8644.550380] LNetError: 281808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8644.565850] LNet: Removed LNI 192.168.202.110@tcp [ 8645.544447] Key type .llcrypt unregistered [ 8645.546115] Key type ._llcrypt unregistered [ 8659.247320] Key type ._llcrypt registered [ 8659.250367] Key type .llcrypt registered [ 8659.373298] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8680.141099] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 8700.470760] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8702.489931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8702.606025] alg: No test for adler32 (adler32-zlib) [ 8703.644821] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 8703.827832] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8705.567247] Key type lgssc registered [ 8707.006765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8749.080437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8749.119709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8750.492090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8750.547239] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8750.693461] Lustre: lustre-MDT0000: new disk, initializing [ 8750.933217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8750.963166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8756.273485] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8768.724786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8768.869685] Lustre: 285359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8768.905374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8768.909045] Lustre: Skipped 1 previous similar message [ 8769.011933] Lustre: lustre-MDT0001: new disk, initializing [ 8769.177426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8769.261919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8769.281639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8775.951838] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8787.611371] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8794.576443] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8803.569536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8803.915370] Lustre: lustre-OST0000: new disk, initializing [ 8803.918898] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8803.929536] Lustre: 287334:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8804.028668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8812.303616] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8812.761947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8812.781504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8812.992289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8824.436815] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8829.968627] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8830.191324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8835.290537] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8835.569933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8838.630847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8838.643159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8838.660735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8838.668953] Lustre: Skipped 1 previous similar message [ 8843.911874] Lustre: server umount lustre-OST0000 complete [ 8848.865694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8848.879442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8848.880175] Lustre: Skipped 2 previous similar messages [ 8848.890097] Lustre: Skipped 1 previous similar message [ 8853.985176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8853.993632] Lustre: Skipped 1 previous similar message [ 8854.380641] Lustre: server umount lustre-MDT0000 complete [ 8858.635695] LustreError: 286155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852028 with bad export cookie 11691773552575496409 [ 8858.638039] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8858.657041] LustreError: 286155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8859.158720] Lustre: server umount lustre-MDT0001 complete [ 8870.842045] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:20:38 (1788852038) [ 8880.178188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8880.664545] LustreError: 289747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8880.682085] LustreError: 289747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8880.724811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8885.728210] LustreError: 289748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8885.950800] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8890.849363] LustreError: 289747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8895.673854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8896.079375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8900.585049] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8908.847320] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8917.750741] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8933.424094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8934.188634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8946.654956] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8958.282568] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8971.239704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8971.245762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8971.273121] LustreError: Skipped 1 previous similar message [ 8971.275515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8975.173237] Lustre: server umount lustre-OST0000 complete [ 8981.497884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8981.510074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8981.529915] Lustre: Skipped 2 previous similar messages [ 8981.538298] Lustre: Skipped 1 previous similar message [ 8987.051907] Lustre: server umount lustre-MDT0000 complete [ 8991.719311] LustreError: 289744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8991.762185] LustreError: 289744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8992.388580] LustreError: 292534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852162 with bad export cookie 11691773552575497291 [ 8992.389656] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8992.409123] LustreError: 292534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8993.194333] Lustre: server umount lustre-MDT0001 complete [ 9004.969829] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 9007.131663] Key type lgssc unregistered [ 9007.619894] LNet: 293285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9007.627143] LNetError: 293285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9007.644274] LNet: Removed LNI 192.168.202.110@tcp [ 9008.758763] Key type .llcrypt unregistered [ 9008.761086] Key type ._llcrypt unregistered [ 9039.281382] Key type ._llcrypt registered [ 9039.285955] Key type .llcrypt registered [ 9039.384216] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 9055.319385] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9056.464201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9056.497994] alg: No test for adler32 (adler32-zlib) [ 9057.817090] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 9058.040768] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 9059.744241] Key type lgssc registered [ 9061.000343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9112.243644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9112.299481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9113.775584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9113.836833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9113.938632] Lustre: lustre-MDT0000: new disk, initializing [ 9114.033113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9114.049636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9120.268527] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9131.645579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9131.755856] Lustre: 297111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9131.789690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9131.792112] Lustre: Skipped 1 previous similar message [ 9131.837488] Lustre: lustre-MDT0001: new disk, initializing [ 9131.893602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9131.913894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9131.921863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9135.918283] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9146.030721] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9153.466391] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9160.739162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9161.066097] Lustre: lustre-OST2710: new disk, initializing [ 9161.071479] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9161.077862] Lustre: 299081:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9161.205579] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9167.807761] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9170.528173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9170.539293] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9170.715604] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9177.889103] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9185.573686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9185.704685] Lustre: lustre-OST03e8: new disk, initializing [ 9185.706804] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9185.710164] Lustre: 300143:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9185.767533] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9193.025959] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9195.691478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9195.707529] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9195.830383] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9200.406417] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9207.802250] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9207.979423] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9213.391536] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9213.551608] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9218.113731] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9218.571062] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9224.147117] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9224.673863] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9231.840951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9231.868361] Lustre: Skipped 1 previous similar message [ 9231.869759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9231.885457] Lustre: Skipped 1 previous similar message [ 9236.651365] Lustre: server umount lustre-MDT0000 complete [ 9240.110238] LustreError: 297101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852409 with bad export cookie 13606916397917071983 [ 9240.118808] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9240.129897] LustreError: 297101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9240.599846] Lustre: server umount lustre-MDT0001 complete [ 9255.911724] Lustre: server umount lustre-OST2710 complete [ 9257.253381] Lustre: 294905:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788852411/real 1788852411] req@000000008a1976e3 x1875747512848256/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788852427 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9257.296158] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9257.306689] Lustre: Skipped 2 previous similar messages [ 9260.730720] Lustre: server umount lustre-OST03e8 complete [ 9282.484804] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 9294.539955] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9340.870644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9341.051414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9341.088435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9341.133614] Lustre: lustre-MDT0000: new disk, initializing [ 9341.189707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9341.209966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9346.395171] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9359.090071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9359.350362] Lustre: 305395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9359.474638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9359.481363] Lustre: Skipped 1 previous similar message [ 9359.582868] Lustre: lustre-MDT0001: new disk, initializing [ 9359.806221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9359.894773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9359.913619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9365.749668] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9379.547745] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9387.062454] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9394.671607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9394.996455] Lustre: lustre-OST0000: new disk, initializing [ 9395.000717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9395.013450] Lustre: 307362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9395.171443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9401.332153] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9403.994987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9404.007508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9404.145258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9411.219556] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9416.468460] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9416.721298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9421.137174] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9421.384904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9424.865335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9424.870089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9424.870684] LustreError: Skipped 1 previous similar message [ 9424.871890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9424.888604] Lustre: Skipped 1 previous similar message [ 9429.581217] Lustre: server umount lustre-OST0000 complete [ 9435.109742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9435.126874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9435.146662] Lustre: Skipped 1 previous similar message [ 9435.175076] Lustre: Skipped 2 previous similar messages [ 9440.036604] Lustre: server umount lustre-MDT0000 complete [ 9440.225181] LustreError: 305404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9440.265133] LustreError: 305404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9444.332750] LustreError: 305405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9444.357438] LustreError: 305405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9444.559771] LustreError: 305387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852614 with bad export cookie 13606916397917073677 [ 9444.562021] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9444.569252] LustreError: 305387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9445.214351] Lustre: server umount lustre-MDT0001 complete [ 9456.440963] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:30:23 (1788852623) [ 9458.543390] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9460.815254] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:30:28 (1788852628) [ 9465.359374] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9496.856062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9497.178247] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9497.194828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9497.418586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9503.169944] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9513.048493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9513.169853] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9513.200338] Lustre: 311220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9517.914887] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9539.920557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9540.459552] Lustre: Denying initial registration attempt from nid 192.168.202.110@tcp, specified as failover [ 9540.473598] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9540.499142] LustreError: 312677:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9540.511825] LustreError: 312677:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9540.522596] LustreError: 312677:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9540.629179] Lustre: server umount lustre-OST0000 complete [ 9540.631094] LustreError: 312677:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9553.380762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9553.388236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9553.417865] Lustre: Skipped 1 previous similar message [ 9553.434084] Lustre: Skipped 1 previous similar message [ 9555.990491] Lustre: server umount lustre-MDT0000 complete [ 9558.497226] LustreError: 311232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9558.521380] LustreError: 311232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9560.813018] LustreError: 311212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852730 with bad export cookie 13606916397917074559 [ 9560.814194] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9560.838456] LustreError: 311212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9561.749348] Lustre: server umount lustre-MDT0001 complete [ 9575.613265] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 9579.191590] Key type lgssc unregistered [ 9579.599050] LNet: 313798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9579.614096] LNetError: 313798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9579.644417] LNet: Removed LNI 192.168.202.110@tcp [ 9580.729236] Key type .llcrypt unregistered [ 9580.731609] Key type ._llcrypt unregistered [ 9594.582703] Key type ._llcrypt registered [ 9594.586733] Key type .llcrypt registered [ 9594.677540] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9595.951690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9596.018068] alg: No test for adler32 (adler32-zlib) [ 9597.141078] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 9597.535755] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 9599.223984] Key type lgssc registered [ 9601.683645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9662.213371] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9673.055702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9673.088768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9674.390587] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9674.448590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9674.746157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9681.465358] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9692.161770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9692.325300] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9692.364864] Lustre: 316566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9692.800034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9699.109693] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9707.786864] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9714.988945] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9737.395701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9737.551738] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9737.736326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9745.034814] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9753.387733] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9758.697081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9758.700539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9758.709401] LustreError: Skipped 1 previous similar message [ 9758.711039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9758.763038] Lustre: Skipped 1 previous similar message [ 9762.279718] Lustre: server umount lustre-OST0000 complete [ 9768.932880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9768.935834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9768.958372] Lustre: Skipped 1 previous similar message [ 9768.979814] Lustre: Skipped 2 previous similar messages [ 9773.640440] Lustre: server umount lustre-MDT0000 complete [ 9778.671223] LustreError: 317305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852948 with bad export cookie 17725937122019219838 [ 9778.677175] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9778.698885] LustreError: 317305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9779.493828] Lustre: server umount lustre-MDT0001 complete [ 9794.700715] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 9797.942508] Key type lgssc unregistered [ 9798.421397] LNet: 320141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9798.440809] LNetError: 320141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9799.467440] LNet: Removed LNI 192.168.202.110@tcp [ 9800.636688] Key type .llcrypt unregistered [ 9800.644803] Key type ._llcrypt unregistered [ 9834.696430] Key type ._llcrypt registered [ 9834.703667] Key type .llcrypt registered [ 9834.865635] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9836.982340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9837.110697] alg: No test for adler32 (adler32-zlib) [ 9838.329928] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 9838.711227] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 9840.503184] Key type lgssc registered [ 9842.753507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9851.865583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9851.900091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9853.623229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9857.928414] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9864.178225] LustreError: 321556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9864.208384] LustreError: 321556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9867.513917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9868.080474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9872.428880] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9880.161036] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9888.648419] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9899.513148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9900.324817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9907.992843] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9916.592822] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9938.374946] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 9939.426363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9939.443154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9939.444872] Lustre: Skipped 2 previous similar messages [ 9939.466878] Lustre: Skipped 2 previous similar messages [ 9941.755236] Lustre: server umount lustre-MDT0000 complete [ 9943.450359] LustreError: 324049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9944.546057] LustreError: 321555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9944.563678] LustreError: 321555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9946.036418] LustreError: 321536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853115 with bad export cookie 6078849270037558657 [ 9946.038796] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9946.046128] LustreError: 321536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9948.585312] LustreError: 321552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9948.591193] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.10@tcp (stopping) [ 9949.667627] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9951.714986] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9951.729717] Lustre: Skipped 1 previous similar message [ 9952.510707] Lustre: server umount lustre-MDT0001 complete [ 9959.592206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9968.347692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9970.527481] LustreError: 325160:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9970.534346] LustreError: 325160:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000315cfc73 x1875748331760768/t0(0) o250->MGC192.168.202.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788853140 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9970.579077] LustreError: 325160:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9971.763922] LustreError: 323390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9971.786188] LustreError: 323390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9971.904453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9972.009463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9979.170263] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9981.095835] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9981.098626] LustreError: 325172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9981.144549] LustreError: 325172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9992.857325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9993.553805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9998.870870] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9999.808178] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10011.089862] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10013.868431] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10021.448622] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10023.859489] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10033.657840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10033.677733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10033.710304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10034.663843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10037.633545] Lustre: server umount lustre-OST0000 complete [10044.896391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10044.903078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10044.911923] Lustre: Skipped 1 previous similar message [10044.919823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10044.949849] Lustre: Skipped 3 previous similar messages [10047.949433] Lustre: server umount lustre-MDT0000 complete [10050.019552] LustreError: 325177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10050.041538] LustreError: 325177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10052.387755] LustreError: 321535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853222 with bad export cookie 6078849270037566189 [10052.399446] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10052.882589] Lustre: server umount lustre-MDT0001 complete [10065.146209] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [10068.036577] Key type lgssc unregistered [10068.509088] LNet: 327847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10068.513591] LNetError: 327847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10068.530956] LNet: Removed LNI 192.168.202.110@tcp [10069.355168] Key type .llcrypt unregistered [10069.369565] Key type ._llcrypt unregistered [10105.115862] Key type ._llcrypt registered [10105.117656] Key type .llcrypt registered [10105.193768] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [10106.084921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10106.100556] alg: No test for adler32 (adler32-zlib) [10107.494556] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [10107.803737] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [10109.495723] Key type lgssc registered [10111.186209] Lustre: Echo OBD driver; http://www.lustre.org/ [10120.938866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10120.968956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10122.728787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10127.709759] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10132.971158] LustreError: 329263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10133.015570] LustreError: 329263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10138.087476] LustreError: 329264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10141.935076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10142.239660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10148.068488] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10156.421445] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10163.779987] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10167.797840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10167.813479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10167.822647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10171.876251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10171.878048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10175.942754] Lustre: server umount lustre-MDT0000 complete [10176.994938] LustreError: 329264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10177.006417] LustreError: 329264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10180.337922] LustreError: 329243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853350 with bad export cookie 10389741041383690120 [10180.344392] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10180.743754] Lustre: server umount lustre-MDT0001 complete [10201.153344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10201.402466] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10201.441914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10201.638658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10207.222789] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10217.501549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10217.591777] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10217.614142] Lustre: 331856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10217.856701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10223.412905] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10232.909158] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10239.374605] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10250.748106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10251.130111] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10251.147931] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10251.171334] LustreError: 333740:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [10251.185736] LustreError: 333740:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10251.190442] LustreError: 333740:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10251.307102] Lustre: server umount lustre-OST0000 complete [10251.312630] LustreError: 333740:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10263.361206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10263.601430] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10263.826503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10270.882886] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10273.375709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10279.858806] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10292.011894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10292.183822] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10292.282335] Lustre: lustre-OST0001: new disk, initializing [10292.289200] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10292.303269] Lustre: 335282:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10292.391797] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10301.750651] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10302.087746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10302.107749] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10302.386967] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10312.436348] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10317.793518] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10317.807861] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10317.816303] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10317.841442] Lustre: Skipped 1 previous similar message [10317.856068] Lustre: Skipped 2 previous similar messages [10322.776181] Lustre: server umount lustre-OST0001 complete [10322.924319] LustreError: 334206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10328.049336] LustreError: 334207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10328.072130] LustreError: 334207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10329.568823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10329.575161] LustreError: Skipped 1 previous similar message [10329.584106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10329.607330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10338.280262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10338.283329] LustreError: 336085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10338.291174] Lustre: Skipped 4 previous similar messages [10338.303899] LustreError: 336085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10342.367180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10342.653625] Lustre: server umount lustre-OST0000 complete [10348.513325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10348.523973] Lustre: Skipped 1 previous similar message [10348.529227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10348.535154] Lustre: Skipped 1 previous similar message [10352.405724] Lustre: server umount lustre-MDT0000 complete [10356.528500] LustreError: 331848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853526 with bad export cookie 10389741041383690666 [10356.530111] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10356.540575] LustreError: 331848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10357.084477] Lustre: server umount lustre-MDT0001 complete [10369.914128] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [10373.278856] Key type lgssc unregistered [10373.605640] LNet: 337241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10373.611481] LNetError: 337241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10373.621346] LNet: Removed LNI 192.168.202.110@tcp [10374.501662] Key type .llcrypt unregistered [10374.503557] Key type ._llcrypt unregistered [10454.373032] Key type ._llcrypt registered [10454.379672] Key type .llcrypt registered [10454.570234] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [10470.790153] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [10472.431507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10472.473927] alg: No test for adler32 (adler32-zlib) [10473.560638] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [10473.897936] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [10475.663195] Key type lgssc registered [10476.898821] Lustre: Echo OBD driver; http://www.lustre.org/ [10513.000125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10513.061138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10514.404919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10514.442532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10514.506203] Lustre: lustre-MDT0000: new disk, initializing [10514.600262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10514.611219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10519.060920] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10531.529239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10531.677975] Lustre: 342173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10531.732906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10531.737899] Lustre: Skipped 1 previous similar message [10531.836886] Lustre: lustre-MDT0001: new disk, initializing [10532.019446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10532.114770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10532.133793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10536.990761] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10547.126457] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10555.295582] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10566.808759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10567.196975] Lustre: lustre-OST0000: new disk, initializing [10567.199574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10567.202852] Lustre: 344144:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10567.250555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10573.263698] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10576.065907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10576.080062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10576.231668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10585.511133] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10590.968931] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10591.361237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10596.459822] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10596.648704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10601.956971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10601.961460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10601.976032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10601.981089] Lustre: Skipped 1 previous similar message [10604.768736] Lustre: server umount lustre-OST0000 complete [10612.193064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10612.210401] Lustre: Skipped 1 previous similar message [10612.214303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10612.232186] Lustre: Skipped 1 previous similar message [10616.000324] Lustre: server umount lustre-MDT0000 complete [10620.381888] LustreError: 342165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853790 with bad export cookie 15020372682103745705 [10620.387317] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10620.405045] LustreError: 342165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10620.678037] Lustre: server umount lustre-MDT0001 complete [10628.891545] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:49:56 (1788853796) [10639.541381] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:50:06 (1788853806) [10650.510529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10651.060267] LustreError: 346899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10651.076862] LustreError: 346899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10651.166505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10656.227314] LustreError: 346900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10656.759652] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10660.321568] LustreError: 346899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10665.444513] LustreError: 346900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10665.918028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10666.455118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10671.565417] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10679.723339] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10687.524369] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10696.187342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10696.455267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10703.875754] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10711.995303] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10731.894189] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10734.672072] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10736.884994] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10739.979732] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10742.570836] Lustre: Failing over lustre-MDT0000 [10742.910935] Lustre: server umount lustre-MDT0000 complete [10743.264215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10743.282490] LustreError: Skipped 1 previous similar message [10743.291564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10743.318643] Lustre: Skipped 1 previous similar message [10743.324606] LustreError: 347884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10746.849808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10746.878073] Lustre: Skipped 1 previous similar message [10751.970280] LustreError: 346895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10751.983399] LustreError: 346895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10762.209073] Lustre: 340395:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788853916/real 1788853916] req@000000001e946580 x1875748997640960/t0(0) o400->MGC192.168.202.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788853932 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10762.274874] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10764.614042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10771.349490] LustreError: 346894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10771.366526] LustreError: 346894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [10773.818161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10773.890994] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10776.473846] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10778.969626] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10779.116237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10779.154866] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10779.191793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10789.699454] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10791.671417] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10795.795917] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10797.767244] Lustre: Failing over lustre-MDT0000 [10798.065565] Lustre: server umount lustre-MDT0000 complete [10799.589988] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10799.610814] Lustre: Skipped 1 previous similar message [10804.706610] LustreError: 346899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10804.736275] LustreError: 346899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [10815.452963] Lustre: 340397:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788853969/real 1788853969] req@0000000037e95c10 x1875748997662208/t0(0) o400->MGC192.168.202.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788853985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10815.496113] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10817.894839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10826.060792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10826.120243] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10826.209288] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10831.353951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10831.362443] Lustre: Skipped 2 previous similar messages [10831.415594] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10831.470484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10831.671785] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10841.694627] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10843.636747] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10851.816736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10851.817239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10851.840634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10851.857258] Lustre: Skipped 2 previous similar messages [10851.886394] Lustre: Skipped 2 previous similar messages [10856.114670] Lustre: server umount lustre-MDT0000 complete [10859.540564] LustreError: 346880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854029 with bad export cookie 15020372682103748421 [10859.545278] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10859.551710] LustreError: 346880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10859.783738] Lustre: server umount lustre-MDT0001 complete [10874.748771] Lustre: server umount lustre-OST0000 complete [10893.311737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10893.885986] LustreError: 353266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10893.900859] LustreError: 353266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [10893.972217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10898.755136] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10907.069656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10912.171199] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10920.568135] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10928.229256] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10938.258107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10944.906363] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10953.642675] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10964.216296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10971.973464] Lustre: server umount lustre-OST0000 complete [10979.296728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10979.305788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10979.322741] Lustre: Skipped 1 previous similar message [10979.326127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10984.931765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10984.940176] Lustre: Skipped 2 previous similar messages [10986.041864] Lustre: server umount lustre-MDT0000 complete [10989.691469] LustreError: 353246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854159 with bad export cookie 15020372682103749163 [10989.709242] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10999.536037] Lustre: server umount lustre-MDT0001 complete [11013.531481] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [11016.421333] Key type lgssc unregistered [11016.826575] LNet: 356820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11016.831472] LNetError: 356820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11017.897636] LNet: Removed LNI 192.168.202.110@tcp [11019.315816] Key type .llcrypt unregistered [11019.319779] Key type ._llcrypt unregistered [11055.028249] Key type ._llcrypt registered [11055.034196] Key type .llcrypt registered [11055.155591] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11056.489874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11056.531454] alg: No test for adler32 (adler32-zlib) [11057.567302] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [11057.889560] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [11059.511233] Key type lgssc registered [11060.601830] Lustre: Echo OBD driver; http://www.lustre.org/ [11069.552937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11069.573291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11070.940036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11075.325349] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11081.186682] LustreError: 358338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11081.211419] LustreError: 358338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11083.246148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11083.670481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11088.498533] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11097.029948] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11104.212794] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11111.883454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11112.340770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11116.476052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11119.970702] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11131.536747] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11149.283421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11149.292181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11149.293200] Lustre: Skipped 1 previous similar message [11149.305658] Lustre: Skipped 2 previous similar messages [11151.907925] Lustre: server umount lustre-MDT0000 complete [11154.409219] LustreError: 358333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11154.430340] LustreError: 358333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11156.194425] LustreError: 358318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854326 with bad export cookie 8287866219001317042 [11156.206347] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11156.210817] LustreError: 358318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11156.626494] Lustre: server umount lustre-MDT0001 complete [11171.307431] Lustre: server umount lustre-OST0000 complete [11188.135598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11188.959390] LustreError: 362073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11189.092440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11193.374094] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11194.338461] LustreError: 362074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11194.367717] LustreError: 362074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11198.443773] LustreError: 362073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11202.984710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11203.521475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11208.786823] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11216.448977] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11222.184431] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11229.189639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11229.473808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11233.599763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11235.557436] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11243.615721] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11259.373628] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11259.377435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11259.424031] Lustre: Skipped 2 previous similar messages [11259.436750] Lustre: Skipped 2 previous similar messages [11263.405518] Lustre: server umount lustre-MDT0000 complete [11264.520443] LustreError: 362073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11264.531993] LustreError: 362073:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11268.768701] LustreError: 362056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854438 with bad export cookie 8287866219001318155 [11268.777570] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11268.791948] LustreError: 362056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11269.175231] Lustre: server umount lustre-MDT0001 complete [11284.631879] Lustre: server umount lustre-OST0000 complete [11303.693214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11304.295247] LustreError: 365997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11304.428275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11309.319834] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11317.829051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11323.217357] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11332.069816] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11339.896905] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11348.759153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11349.045878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11349.050217] Lustre: Skipped 1 previous similar message [11350.070197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11357.023587] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11367.784632] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11377.636367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11377.639172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11377.653367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11377.659615] Lustre: Skipped 2 previous similar messages [11382.108547] Lustre: server umount lustre-OST0000 complete [11387.879627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11387.892480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11387.898539] Lustre: Skipped 1 previous similar message [11392.548487] Lustre: server umount lustre-MDT0000 complete [11393.009304] LustreError: 365994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11393.026405] LustreError: 365994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11396.377042] LustreError: 365979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854566 with bad export cookie 8287866219001319177 [11396.384227] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11396.385302] LustreError: 365979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11397.346583] Lustre: server umount lustre-MDT0001 complete [11411.119722] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [11413.914187] Key type lgssc unregistered [11414.139876] LNet: 369535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11414.144973] LNetError: 369535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11415.210653] LNet: Removed LNI 192.168.202.110@tcp [11415.753197] Key type .llcrypt unregistered [11415.754872] Key type ._llcrypt unregistered [11448.952428] Key type ._llcrypt registered [11448.954844] Key type .llcrypt registered [11449.084770] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11450.335102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11450.368934] alg: No test for adler32 (adler32-zlib) [11451.624528] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [11451.836567] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [11453.623583] Key type lgssc registered [11455.087734] Lustre: Echo OBD driver; http://www.lustre.org/ [11464.508982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11464.524666] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11464.539076] LustreError: 370974:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11464.562971] LustreError: 370974:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11464.575209] LustreError: 370974:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11464.580191] LustreError: 370974:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11464.591200] LustreError: 370974:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11474.293351] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11474.296815] LustreError: 371373:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11474.306747] LustreError: 371373:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11474.309409] LustreError: 371373:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11474.311801] LustreError: 371373:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11474.314862] LustreError: 371373:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11497.640867] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [11500.990333] Key type lgssc unregistered [11501.428501] LNet: 372375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11501.447893] LNetError: 372375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11501.483264] LNet: Removed LNI 192.168.202.110@tcp [11502.513989] Key type .llcrypt unregistered [11502.519947] Key type ._llcrypt unregistered [11522.162541] Key type ._llcrypt registered [11522.163941] Key type .llcrypt registered [11522.220317] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [11540.142738] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11541.591085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11541.625130] alg: No test for adler32 (adler32-zlib) [11542.679632] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [11542.908103] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [11544.591195] Key type lgssc registered [11545.491761] Lustre: Echo OBD driver; http://www.lustre.org/ [11586.526086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11586.615742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11588.114195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11588.143106] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11588.294456] Lustre: lustre-MDT0000: new disk, initializing [11588.358604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11588.370350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11594.049970] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11605.890988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11606.003180] Lustre: 375528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11606.032496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11606.035150] Lustre: Skipped 1 previous similar message [11606.084441] Lustre: lustre-MDT0001: new disk, initializing [11606.182913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11606.228638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11606.243993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11611.294657] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11622.607427] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11628.977312] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11637.484814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11637.885770] Lustre: lustre-OST0000: new disk, initializing [11637.888159] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11637.896159] Lustre: 377498:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11638.015430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11644.099800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11644.115972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11644.243700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11646.301742] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11659.649564] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11666.163401] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11666.570774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11672.042949] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11672.461685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11675.106628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11675.109974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11675.118103] LustreError: Skipped 1 previous similar message [11675.121902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11675.152064] Lustre: Skipped 1 previous similar message [11680.228717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11680.243516] Lustre: Skipped 2 previous similar messages [11680.781559] Lustre: server umount lustre-OST0000 complete [11688.417926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11688.433329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11688.482659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11690.468297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11690.490461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11690.497562] Lustre: Skipped 1 previous similar message [11692.038784] Lustre: server umount lustre-MDT0000 complete [11697.110418] LustreError: 376333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854866 with bad export cookie 6132964363606126517 [11697.123499] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11697.132860] LustreError: 376333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11697.689611] Lustre: server umount lustre-MDT0001 complete [11708.078514] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11710.331486] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:07:57 (1788854877) [11719.011739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11719.726209] LustreError: 379961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11719.745166] LustreError: 379961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11719.802699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11725.294306] LustreError: 379962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11725.422773] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11729.377543] LustreError: 379961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11734.509436] LustreError: 379962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11735.315729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11736.059860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11740.711701] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11751.116309] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11759.306064] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11767.735247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11768.214459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11775.310371] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11783.860234] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11791.870139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11792.084325] Lustre: lustre-OST0001: new disk, initializing [11792.088788] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11792.096898] Lustre: 382799:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11792.204728] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11797.753666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11797.772295] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11797.935681] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11800.149225] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11811.810718] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11818.470777] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11818.494083] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11818.504906] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11818.509735] LustreError: Skipped 1 previous similar message [11818.535823] Lustre: Skipped 1 previous similar message [11822.508873] Lustre: server umount lustre-OST0001 complete [11823.595349] LustreError: 381785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11823.649457] LustreError: 381785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11833.834404] LustreError: 383076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11833.888992] LustreError: 383076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11844.069855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11844.096347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11844.101638] Lustre: Skipped 2 previous similar messages [11849.467735] Lustre: server umount lustre-OST0000 complete [11858.918033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11858.920455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11858.931948] Lustre: Skipped 1 previous similar message [11862.354650] Lustre: server umount lustre-MDT0000 complete [11864.553545] LustreError: 380689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11864.567961] LustreError: 380689:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11867.147558] LustreError: 379942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855036 with bad export cookie 6132964363606127399 [11867.156587] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11867.172026] LustreError: 379942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11873.852623] Lustre: server umount lustre-MDT0001 complete [11888.877756] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [11892.635122] Key type lgssc unregistered [11893.004936] LNet: 384792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11893.013893] LNetError: 384792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11893.026346] LNet: Removed LNI 192.168.202.110@tcp [11894.317525] Key type .llcrypt unregistered [11894.323095] Key type ._llcrypt unregistered [11958.199636] Key type ._llcrypt registered [11958.201495] Key type .llcrypt registered [11958.281995] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11960.132091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11960.155262] alg: No test for adler32 (adler32-zlib) [11961.329449] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [11961.761281] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [11963.495228] Key type lgssc registered [11965.084234] Lustre: Echo OBD driver; http://www.lustre.org/ [11974.062888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11974.096932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11975.478856] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11975.497462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11975.706569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11980.487653] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11990.907079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11990.973309] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11991.031385] Lustre: 387587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11991.443201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11997.697289] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12007.298714] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12013.467559] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12017.121391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12017.128992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12017.152089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12019.682401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12019.686347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12019.698582] Lustre: Skipped 1 previous similar message [12022.000915] Lustre: server umount lustre-MDT0000 complete [12025.777236] LustreError: 387580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855195 with bad export cookie 14551894045617847838 [12025.787778] LustreError: 387580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12025.795287] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12026.252560] Lustre: server umount lustre-MDT0001 complete [12038.540964] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:13:25 (1788855205) [12047.729223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12048.403618] LustreError: 390102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12048.414511] LustreError: 390102:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12048.465472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12053.477676] LustreError: 390103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12054.598415] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12058.593312] LustreError: 390102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12063.717080] LustreError: 390103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12066.462713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12067.107252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12073.472435] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12081.371760] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12089.192645] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12097.416668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12097.554485] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12097.913632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12107.738506] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12117.503760] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12126.749721] Lustre: Permanently deactivating lustre-OST0000 [12126.760428] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12135.026765] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12137.471522] LustreError: 393091:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12137.496184] LustreError: 393091:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12137.515504] LustreError: 393091:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12141.704982] LustreError: 393140:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12141.709113] LustreError: 393140:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12150.086960] Lustre: server umount lustre-OST0000 complete [12153.163295] LustreError: 393391:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12153.174349] LustreError: 393391:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12159.458473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12159.475219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12159.500429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12165.612664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12165.640865] Lustre: Skipped 3 previous similar messages [12165.803774] Lustre: server umount lustre-MDT0000 complete [12170.155148] LustreError: 391938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855339 with bad export cookie 14551894045617848419 [12170.160513] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12170.177995] LustreError: 391938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12179.493251] Lustre: server umount lustre-MDT0001 complete [12190.572688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12197.271212] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12200.667496] LustreError: 394550:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12208.512951] Lustre: 394889:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.110@tcp [12211.281847] Lustre: server umount MGS complete [12226.536093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12227.286526] LustreError: 395614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12227.320908] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12227.330082] Lustre: Skipped 1 previous similar message [12227.453658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12234.125512] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12237.800591] LustreError: 395615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12237.847066] LustreError: 395615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12245.794087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12246.209430] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12246.307671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12252.235885] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12261.705639] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12268.333788] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12271.292724] Lustre: Permanently reactivating lustre-OST0000 [12271.308262] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12271.312881] Lustre: Skipped 2 previous similar messages [12299.743947] LustreError: 397385:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12309.947055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12310.424503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12319.106846] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12330.070851] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12340.199621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12340.201689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12340.231853] Lustre: Skipped 1 previous similar message [12340.248594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12340.252654] LustreError: Skipped 1 previous similar message [12340.268837] Lustre: Skipped 1 previous similar message [12342.308586] Lustre: server umount lustre-OST0000 complete [12348.899252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12348.903114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12348.927222] Lustre: Skipped 1 previous similar message [12348.949250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12353.778059] Lustre: server umount lustre-MDT0000 complete [12355.555475] LustreError: 397444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12355.585892] LustreError: 397444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12358.414394] LustreError: 397189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855528 with bad export cookie 14551894045617849469 [12358.417049] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12358.430429] LustreError: 397189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12359.195822] Lustre: server umount lustre-MDT0001 complete [12378.309659] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12382.025538] Key type lgssc unregistered [12382.351234] LNet: 399946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12382.357548] LNetError: 399946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12382.385289] LNet: Removed LNI 192.168.202.110@tcp [12383.945817] Key type .llcrypt unregistered [12383.950803] Key type ._llcrypt unregistered [12405.865775] Key type ._llcrypt registered [12405.867743] Key type .llcrypt registered [12406.004799] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [12425.600341] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12427.020363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12427.092099] alg: No test for adler32 (adler32-zlib) [12428.378311] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [12428.765109] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12430.543288] Key type lgssc registered [12431.983157] Lustre: Echo OBD driver; http://www.lustre.org/ [12475.448467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12475.482708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12476.839682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12476.901656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12477.004901] Lustre: lustre-MDT0000: new disk, initializing [12477.088420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12477.103259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12482.352848] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12495.306532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12495.503117] Lustre: 403108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12495.545922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12495.549349] Lustre: Skipped 1 previous similar message [12495.600211] Lustre: lustre-MDT0001: new disk, initializing [12495.661652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12495.711739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12495.729950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12501.205309] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12512.405828] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12519.655371] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12527.462649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12527.966442] Lustre: lustre-OST0000: new disk, initializing [12527.977967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12527.996307] Lustre: 405078:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12528.209638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12533.361938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12533.375531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12533.534604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12535.330369] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12548.529625] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12553.350594] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12553.526520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12558.337363] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12558.570285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12564.456989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12564.469329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12564.471766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12566.920064] Lustre: server umount lustre-OST0000 complete [12574.691134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12574.695813] Lustre: Skipped 1 previous similar message [12574.700986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12574.705510] Lustre: Skipped 2 previous similar messages [12579.616747] Lustre: server umount lustre-MDT0000 complete [12584.470572] LustreError: 403100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855754 with bad export cookie 9074163673189838491 [12584.474326] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12584.506152] LustreError: 403100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12585.321604] Lustre: server umount lustre-MDT0001 complete [12596.642667] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12598.951130] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:22:46 (1788855766) [12608.527588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12609.139236] LustreError: 407541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12609.156091] LustreError: 407541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12609.215940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12614.625476] LustreError: 407542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12615.080295] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12618.722217] LustreError: 407541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12623.841705] LustreError: 407542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12625.575515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12626.012257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12631.640376] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12640.731743] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12647.109358] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12655.350790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12655.777825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12663.191803] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12671.356444] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12673.556903] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12681.085156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12688.872088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12688.879810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12688.906436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12688.930258] Lustre: Skipped 2 previous similar messages [12688.942831] LustreError: Skipped 1 previous similar message [12693.987854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12693.994060] Lustre: Skipped 2 previous similar messages [12694.489540] Lustre: server umount lustre-OST0000 complete [12702.688819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12702.694541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12702.711119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12705.882853] Lustre: server umount lustre-MDT0000 complete [12709.344706] LustreError: 407538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12709.354600] LustreError: 407538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12711.293048] LustreError: 407523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855881 with bad export cookie 9074163673189839373 [12711.296465] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12711.308404] LustreError: 407523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12711.993579] Lustre: server umount lustre-MDT0001 complete [12726.482215] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12730.205715] Key type lgssc unregistered [12730.677940] LNet: 411268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12730.695970] LNetError: 411268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12731.750514] LNet: Removed LNI 192.168.202.110@tcp [12732.667803] Key type .llcrypt unregistered [12732.669592] Key type ._llcrypt unregistered [12760.881138] Key type ._llcrypt registered [12760.882694] Key type .llcrypt registered [12760.962732] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12761.251299] Key type .llcrypt unregistered [12761.259534] Key type ._llcrypt unregistered [12783.261989] Key type ._llcrypt registered [12783.266381] Key type .llcrypt registered [12783.382613] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12785.042132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12785.073337] alg: No test for adler32 (adler32-zlib) [12786.182206] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [12786.482297] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12788.192195] Key type lgssc registered [12789.776572] Lustre: Echo OBD driver; http://www.lustre.org/ [12798.081571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12798.098829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12799.461116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12804.483435] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12809.699900] LustreError: 413479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12809.715216] LustreError: 413479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12814.817468] LustreError: 413478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12815.943073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12816.553226] LustreError: 413479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12816.596107] LustreError: 413479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12816.622647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12821.987647] LustreError: 413478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12822.068492] LustreError: 413478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12825.098302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12827.304241] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12830.183745] LustreError: 413478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12830.201476] LustreError: 413478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12836.600895] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12840.424750] LustreError: 413479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12840.440391] LustreError: 413479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12845.591287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12845.900879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12851.538573] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12860.897096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12860.916892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12860.944915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12861.432559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12861.443150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12862.208564] Lustre: server umount lustre-OST0000 complete [12871.652130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12871.656989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12871.657744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12871.657749] Lustre: Skipped 1 previous similar message [12871.682032] Lustre: Skipped 1 previous similar message [12873.719577] Lustre: server umount lustre-MDT0000 complete [12876.780067] LustreError: 413475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12876.795603] LustreError: 413475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12879.260689] LustreError: 413459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856049 with bad export cookie 9294335513135757987 [12879.289086] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12879.290509] LustreError: 413459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12879.652633] Lustre: server umount lustre-MDT0001 complete [12891.657053] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12895.348373] Key type lgssc unregistered [12895.679242] LNet: 416627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12895.691106] LNetError: 416627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12895.709963] LNet: Removed LNI 192.168.202.110@tcp [12896.676845] Key type .llcrypt unregistered [12896.679328] Key type ._llcrypt unregistered [12924.833618] Key type ._llcrypt registered [12924.835169] Key type .llcrypt registered [12924.975510] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12926.679486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12926.752441] alg: No test for adler32 (adler32-zlib) [12927.914666] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [12928.180821] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12929.855195] Key type lgssc registered [12931.408917] Lustre: Echo OBD driver; http://www.lustre.org/ [12941.406722] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12941.437535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12957.028686] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12962.143369] LustreError: 418015:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.110@tcp: failed processing log, type 4: rc = -110 [12988.895199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12998.049506] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13011.136826] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13019.884822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13020.561217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13020.661565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13026.483346] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13029.860196] LustreError: 419054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13029.893452] LustreError: 419054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13034.978419] LustreError: 419054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13036.362131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13036.685161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13040.862933] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13051.731979] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13061.472702] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13076.960559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13076.977446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13076.998934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13077.473174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13077.488604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13077.537706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13079.939569] Lustre: server umount lustre-OST0000 complete [13087.719018] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13087.725604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13087.738307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13087.740937] Lustre: Skipped 2 previous similar messages [13091.313395] Lustre: server umount lustre-MDT0000 complete [13093.856950] LustreError: 419059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13093.898554] LustreError: 419059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13095.771331] LustreError: 418023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856265 with bad export cookie 10077057875494297575 [13095.779191] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13095.786348] LustreError: 418023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13096.184447] Lustre: server umount lustre-MDT0001 complete [13110.346353] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13113.913543] Key type lgssc unregistered [13114.271378] LNet: 421589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13114.280940] LNetError: 421589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13115.304757] LNet: Removed LNI 192.168.202.110@tcp [13116.445102] Key type .llcrypt unregistered [13116.447641] Key type ._llcrypt unregistered [13143.889357] Key type ._llcrypt registered [13143.896453] Key type .llcrypt registered [13144.042973] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [13145.253093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13145.395204] alg: No test for adler32 (adler32-zlib) [13146.549983] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [13146.762044] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [13148.447308] Key type lgssc registered [13149.983995] Lustre: Echo OBD driver; http://www.lustre.org/ [13159.862441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13159.897918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13161.776152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13167.587751] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13172.200884] LustreError: 423004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13172.221385] LustreError: 423004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13177.312103] LustreError: 423005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13177.722046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13178.232861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13184.894212] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13194.574324] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13201.937545] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13211.654397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13212.383841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13220.506090] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13231.433404] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13232.861598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13242.793689] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [13243.370203] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13243.400959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13243.412470] Lustre: Skipped 1 previous similar message [13247.166285] Lustre: server umount lustre-MDT0000 complete [13247.890841] LustreError: 423000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13250.111428] LustreError: 423738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.10@tcp arrived at 1788856419 with bad export cookie 8194305068753543287 [13252.560281] LustreError: 423738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856422 with bad export cookie 8194305068753542622 [13252.563451] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13252.584752] LustreError: 423738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13253.600489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13253.602484] LustreError: 423004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13253.609285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13253.609294] Lustre: Skipped 2 previous similar messages [13253.613680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13253.636608] LustreError: 423004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13257.633313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13257.633723] LustreError: 423001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13257.696905] LustreError: 423001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13259.133491] Lustre: server umount lustre-OST0000 complete [13267.604771] Lustre: server umount lustre-MDT0001 complete [13280.831599] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13284.463401] Key type lgssc unregistered [13284.897887] LNet: 426690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13284.913737] LNetError: 426690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13284.943813] LNet: Removed LNI 192.168.202.110@tcp [13285.931980] Key type .llcrypt unregistered [13285.935618] Key type ._llcrypt unregistered [13319.456517] Key type ._llcrypt registered [13319.459240] Key type .llcrypt registered [13319.667284] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [13321.610904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13321.671172] alg: No test for adler32 (adler32-zlib) [13322.850554] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [13323.172134] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [13324.919167] Key type lgssc registered [13326.344752] Lustre: Echo OBD driver; http://www.lustre.org/ [13335.449152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13335.472628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13337.256648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13342.201714] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13346.788110] LustreError: 428104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13346.833319] LustreError: 428104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13351.907296] LustreError: 428105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13353.550344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13354.241995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13359.985127] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13368.470839] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13375.825924] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13383.562978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13384.136038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13395.937733] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13406.724401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13407.225439] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13415.378520] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.10@tcp (stopping) [13415.906925] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13415.914225] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13415.935713] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13416.933536] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13416.956064] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13416.957600] Lustre: Skipped 1 previous similar message [13418.946782] Lustre: server umount lustre-MDT0001 complete [13420.433927] LustreError: 430513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13427.171657] LustreError: 428105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13427.193205] LustreError: 428105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13432.296399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13432.297457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13432.297620] LustreError: 428104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13432.335206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13433.913890] Lustre: server umount lustre-OST0000 complete [13439.208269] Lustre: server umount lustre-MDT0000 complete [13455.897271] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13459.496839] Key type lgssc unregistered [13460.010503] LNet: 431796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13460.018383] LNetError: 431796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13460.046524] LNet: Removed LNI 192.168.202.110@tcp [13460.781499] Key type .llcrypt unregistered [13460.786435] Key type ._llcrypt unregistered [13495.268960] Key type ._llcrypt registered [13495.274233] Key type .llcrypt registered [13495.452780] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [13512.765795] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [13514.047563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13514.248671] alg: No test for adler32 (adler32-zlib) [13515.481134] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [13515.765536] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [13517.519204] Key type lgssc registered [13518.744929] Lustre: Echo OBD driver; http://www.lustre.org/ [13568.232562] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [13587.740506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13587.784720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13589.168839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13589.301584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13589.425455] Lustre: lustre-MDT0000: new disk, initializing [13589.664875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13589.700938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13596.814740] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13616.016632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13616.317247] Lustre: lustre-OST0000: new disk, initializing [13616.325696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13616.333292] Lustre: Skipped 1 previous similar message [13616.341782] Lustre: 436826:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13616.423655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13623.863334] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13624.423353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13624.446642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13624.479092] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13634.773809] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13650.465553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13650.638561] Lustre: 435809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13650.713702] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13650.786517] Lustre: lustre-MDT0001: new disk, initializing [13650.938210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13651.014413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13651.041869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13655.361948] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13667.515887] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13667.672096] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13674.795812] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13675.140585] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13680.616803] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13680.640067] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13683.650762] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.10@tcp (stopping) [13683.656194] Lustre: Skipped 1 previous similar message [13685.731746] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13687.133927] Lustre: server umount lustre-MDT0001 complete [13695.990620] LustreError: 435820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13696.019196] LustreError: 435820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13701.089989] LustreError: 435819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13701.123070] LustreError: 435819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13706.212660] LustreError: 435820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13706.239029] LustreError: 435820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13711.329653] LustreError: 435819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13711.334652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13711.385624] Lustre: Skipped 1 previous similar message [13711.403480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13711.404573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13711.461960] Lustre: Skipped 1 previous similar message [13713.170211] Lustre: server umount lustre-OST0000 complete [13716.450329] LustreError: 435820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13732.319143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13732.726736] Lustre: server umount lustre-MDT0000 complete [13751.328782] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13756.272093] Key type lgssc unregistered [13756.958299] LNet: 440579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13756.973487] LNetError: 440579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13756.991837] LNet: Removed LNI 192.168.202.110@tcp [13758.532361] Key type .llcrypt unregistered [13758.536071] Key type ._llcrypt unregistered [13811.913493] Key type ._llcrypt registered [13811.915568] Key type .llcrypt registered [13812.007662] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13812.190365] Key type .llcrypt unregistered [13812.192113] Key type ._llcrypt unregistered [13834.072362] Key type ._llcrypt registered [13834.073933] Key type .llcrypt registered [13834.167539] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [13835.743428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13835.770418] alg: No test for adler32 (adler32-zlib) [13837.268533] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [13837.825909] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [13839.575296] Key type lgssc registered [13841.177736] Lustre: Echo OBD driver; http://www.lustre.org/ [13874.661064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13874.740269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13876.170993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13876.215418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13876.280221] Lustre: lustre-MDT0000: new disk, initializing [13876.347259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13876.375632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13882.257928] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13893.986076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13894.255511] Lustre: 443957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13894.302332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13894.310451] Lustre: Skipped 1 previous similar message [13894.408313] Lustre: lustre-MDT0001: new disk, initializing [13894.498264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13894.527807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13894.541474] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13899.643914] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13910.398397] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13916.203884] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13924.760470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13925.017071] Lustre: lustre-OST0000: new disk, initializing [13925.023533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13925.032978] Lustre: 445925:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13925.156926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13930.542670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13930.553850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13930.682423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13932.851641] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13944.373359] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13961.185070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13961.204160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13961.205502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13961.211411] Lustre: Skipped 1 previous similar message [13962.752630] Lustre: server umount lustre-MDT0000 complete [13966.305243] LustreError: 443951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857136 with bad export cookie 11181992416418164153 [13966.318081] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13966.323286] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13966.349749] Lustre: Skipped 2 previous similar messages [13966.409553] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13966.434577] Lustre: Skipped 1 previous similar message [13966.794611] Lustre: server umount lustre-MDT0001 complete [13973.181849] Lustre: server umount lustre-OST0000 complete [13986.292893] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:45:53 (1788857153) [13995.676954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13996.142784] LustreError: 447953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13996.156771] LustreError: 447953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13996.221814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14001.633875] LustreError: 447954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14002.553516] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14005.734300] LustreError: 447953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14010.858930] LustreError: 447954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14013.640746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14014.261067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14020.633948] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14031.161420] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14037.788090] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14058.502086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14059.006917] Lustre: Found index 0 for lustre-OST0000, updating log [14059.253408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14065.401034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14067.826204] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14078.279238] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14086.117326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14086.129301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14086.151761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14088.333489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14088.340091] Lustre: Skipped 2 previous similar messages [14091.776242] Lustre: server umount lustre-OST0000 complete [14111.713882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14111.725627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14111.748411] Lustre: Skipped 1 previous similar message [14111.762667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14111.776663] Lustre: Skipped 3 previous similar messages [14116.808925] Lustre: server umount lustre-MDT0000 complete [14117.542310] LustreError: 449806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14117.575324] LustreError: 449806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14121.166474] LustreError: 447934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857290 with bad export cookie 11181992416418165546 [14121.176107] LustreError: 447934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [14121.187684] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14121.699545] Lustre: server umount lustre-MDT0001 complete [14126.623796] LNet: 1 local NIs in recovery (showing 1): 192.168.202.110@tcp [14131.176958] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:48:19 (1788857299) [14764.860882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14765.452049] LustreError: 454368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14765.479347] LustreError: 454368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14765.603557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14770.662291] LustreError: 454369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14772.347924] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14775.779843] LustreError: 454368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14781.920479] LustreError: 454369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14781.947475] LustreError: 454369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14783.779214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14784.138290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14790.144484] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14798.468655] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14804.987369] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14817.951385] LNet: 1 local NIs in recovery (showing 1): 192.168.202.110@tcp [14825.981191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14826.386178] Lustre: Found index 0 for lustre-OST0000, updating log [14826.809953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14834.099462] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14842.916396] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14846.253321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14846.283238] Lustre: Skipped 1 previous similar message [14846.296154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14846.306167] Lustre: Skipped 2 previous similar messages [14851.655208] Lustre: server umount lustre-OST0000 complete [14857.060880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14857.075806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14857.104043] Lustre: Skipped 1 previous similar message [14857.133880] Lustre: Skipped 1 previous similar message [14859.170883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14859.177825] Lustre: Skipped 3 previous similar messages [14862.449727] Lustre: server umount lustre-MDT0000 complete [14863.267177] LustreError: 454369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14863.289941] LustreError: 454369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14867.703861] LustreError: 454349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858037 with bad export cookie 11181992416418166624 [14867.708350] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14867.724074] LustreError: 454349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [14868.294522] Lustre: server umount lustre-MDT0001 complete [14877.346770] LNet: 1 local NIs in recovery (showing 1): 192.168.202.110@tcp [14907.223536] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [14924.428116] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [14972.200822] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:02:19 (1788858139) [14995.876291] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [15000.566622] Key type lgssc unregistered [15001.119059] LNet: 461019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15001.128841] LNetError: 461019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15087.144342] LNet: Removed LNI 192.168.202.110@tcp [15088.276978] Key type .llcrypt unregistered [15088.282335] Key type ._llcrypt unregistered [15111.310777] Key type ._llcrypt registered [15111.313757] Key type .llcrypt registered [15111.506161] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [15112.952695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15113.000293] alg: No test for adler32 (adler32-zlib) [15114.142597] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [15114.407747] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [15116.151864] Key type lgssc registered [15117.801703] Lustre: Echo OBD driver; http://www.lustre.org/ [15123.655300] LNet: Added LNI 192.168.202.110@tcp100 [8/256/0/180] [15124.621616] LNet: Added LNI 192.168.202.110@tcp101 [8/256/0/180] [15125.701273] LNet: Added LNI 192.168.202.110@tcp102 [8/256/0/180] [15126.796222] LNet: Added LNI 192.168.202.110@tcp103 [8/256/0/180] [15127.799775] LNet: Added LNI 192.168.202.110@tcp104 [8/256/0/180] [15128.826395] LNet: Added LNI 192.168.202.110@tcp105 [8/256/0/180] [15129.808548] LNet: Added LNI 192.168.202.110@tcp106 [8/256/0/180] [15130.874197] LNet: Added LNI 192.168.202.110@tcp107 [8/256/0/180] [15131.802141] LNet: Added LNI 192.168.202.110@tcp108 [8/256/0/180] [15132.884087] LNet: Added LNI 192.168.202.110@tcp109 [8/256/0/180] [15133.920570] LNet: Added LNI 192.168.202.110@tcp110 [8/256/0/180] [15134.900594] LNet: Added LNI 192.168.202.110@tcp111 [8/256/0/180] [15136.005714] LNet: Added LNI 192.168.202.110@tcp112 [8/256/0/180] [15136.999498] LNet: Added LNI 192.168.202.110@tcp113 [8/256/0/180] [15137.945301] LNet: Added LNI 192.168.202.110@tcp114 [8/256/0/180] [15139.045970] LNet: Added LNI 192.168.202.110@tcp115 [8/256/0/180] [15140.253765] LNet: Added LNI 192.168.202.110@tcp116 [8/256/0/180] [15141.511681] LNet: Added LNI 192.168.202.110@tcp117 [8/256/0/180] [15142.423943] LNet: Added LNI 192.168.202.110@tcp118 [8/256/0/180] [15143.586449] LNet: Added LNI 192.168.202.110@tcp119 [8/256/0/180] [15144.558401] LNet: Added LNI 192.168.202.110@tcp120 [8/256/0/180] [15145.494703] LNet: Added LNI 192.168.202.110@tcp121 [8/256/0/180] [15146.524980] LNet: Added LNI 192.168.202.110@tcp122 [8/256/0/180] [15147.673387] LNet: Added LNI 192.168.202.110@tcp123 [8/256/0/180] [15148.751869] LNet: Added LNI 192.168.202.110@tcp124 [8/256/0/180] [15149.814564] LNet: Added LNI 192.168.202.110@tcp125 [8/256/0/180] [15151.005049] LNet: Added LNI 192.168.202.110@tcp126 [8/256/0/180] [15152.242197] LNet: Added LNI 192.168.202.110@tcp127 [8/256/0/180] [15153.305642] LNet: Added LNI 192.168.202.110@tcp128 [8/256/0/180] [15154.366507] LNet: Added LNI 192.168.202.110@tcp129 [8/256/0/180] [15155.403871] LNet: Added LNI 192.168.202.110@tcp130 [8/256/0/180] [15156.477398] LNet: Added LNI 192.168.202.110@tcp131 [8/256/0/180] [15157.419794] LNet: Added LNI 192.168.202.110@tcp132 [8/256/0/180] [15158.563624] LNet: Added LNI 192.168.202.110@tcp133 [8/256/0/180] [15159.450995] LNet: Added LNI 192.168.202.110@tcp134 [8/256/0/180] [15160.286692] LNet: Added LNI 192.168.202.110@tcp135 [8/256/0/180] [15167.743907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15167.805741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15169.002052] Lustre: 464045:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [15169.143120] LNetError: 464067:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [15169.160269] LNetError: Protocol error connecting to 192.168.202.110@tcp119 at host 192.168.202.110:988: is it running a compatible version of Lustre? [15169.174159] LNetError: Skipped 3 previous similar messages [15169.286779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15169.322369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15169.453930] Lustre: lustre-MDT0000: new disk, initializing [15169.530143] Lustre: 464101:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [15169.552975] Lustre: 464101:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15169.573929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15169.605630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15169.699076] LNet: 8 local NIs in recovery (showing 8): 192.168.202.110@tcp123, 192.168.202.110@tcp126, 192.168.202.110@tcp120, 192.168.202.110@tcp128, 192.168.202.110@tcp129, 192.168.202.110@tcp127, 192.168.202.110@tcp115, 192.168.202.110@tcp122 [15170.208055] LNetError: 464075:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [15170.231308] LNetError: 464075:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [15170.243636] LNetError: Protocol error connecting to 192.168.202.110@tcp111 at host 192.168.202.110:988: is it running a compatible version of Lustre? [15170.257984] LNetError: Skipped 2 previous similar messages [15172.843783] LNetError: 464069:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [15172.848349] LNetError: 464069:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [15172.851747] LNetError: Protocol error connecting to 192.168.202.110@tcp123 at host 192.168.202.110:988: is it running a compatible version of Lustre? [15172.856784] LNetError: Skipped 10 previous similar messages [15174.576510] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15176.931596] LNetError: 464065:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [15176.955397] LNetError: 464065:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [15176.963856] LNetError: Protocol error connecting to 192.168.202.110@tcp123 at host 192.168.202.110:988: is it running a compatible version of Lustre? [15176.982074] LNetError: Skipped 4 previous similar messages [15184.098102] LNetError: 461500:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [15184.108190] LNetError: 461500:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15184.113770] LNetError: Protocol error connecting to 192.168.202.110@tcp123 at host 192.168.202.110:988: is it running a compatible version of Lustre? [15187.964897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15188.152971] Lustre: 464063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15188.258699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15188.266418] Lustre: Skipped 1 previous similar message [15188.382663] Lustre: lustre-MDT0001: new disk, initializing [15188.413851] Lustre: 464863:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [15188.421943] Lustre: 464863:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15188.530953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15188.566440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15188.576122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15194.138139] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15195.729493] Lustre: 465196:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [15195.741374] Lustre: 465196:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15199.460510] LNetError: 464067:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [15199.472398] LNetError: Protocol error connecting to 192.168.202.110@tcp119 at host 192.168.202.110:988: is it running a compatible version of Lustre? [15204.694905] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15211.528531] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15216.873607] LNetError: 464075:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [15216.897530] LNetError: 464075:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [15216.904829] LNetError: Protocol error connecting to 192.168.202.110@tcp125 at host 192.168.202.110:988: is it running a compatible version of Lustre? [15216.913194] LNetError: Skipped 10 previous similar messages [15219.920543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15220.325881] Lustre: lustre-OST0000: new disk, initializing [15220.329648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15220.335163] Lustre: 466035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15220.427888] Lustre: 466040:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [15220.434668] Lustre: 466040:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15220.487750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15227.316928] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15230.181995] LNet: 1 local NIs in recovery (showing 1): 192.168.202.110@tcp123 [15230.571031] Lustre: 466496:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [15230.589256] Lustre: 466496:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15230.706919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15230.741172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15230.850480] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15237.928827] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15246.311628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15246.320554] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15246.326711] Lustre: Skipped 1 previous similar message [15246.328789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15246.351771] LustreError: Skipped 1 previous similar message [15249.635915] LNetError: 461503:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [15249.645397] LNetError: 461503:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 21 previous similar messages [15249.653388] LNetError: Protocol error connecting to 192.168.202.110@tcp124 at host 192.168.202.110:988: is it running a compatible version of Lustre? [15249.668508] LNetError: Skipped 21 previous similar messages [15251.250445] Lustre: server umount lustre-OST0000 complete [15256.546817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15256.556164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15256.556925] Lustre: Skipped 1 previous similar message [15256.576301] Lustre: Skipped 2 previous similar messages [15261.669941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15261.680822] Lustre: Skipped 1 previous similar message [15261.851219] Lustre: server umount lustre-MDT0000 complete [15267.255722] LustreError: 466045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858437 with bad export cookie 11835621348213800473 [15267.257869] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15267.269874] LustreError: 466045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15267.788958] Lustre: server umount lustre-MDT0001 complete [15282.960778] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [15287.312439] Key type lgssc unregistered [15287.728622] LNet: 467950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15287.737947] LNetError: 467950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15288.799217] LNet: Removed LNI 192.168.202.110@tcp [15288.821301] LNetError: 467950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15288.829932] LNet: Removed LNI 192.168.202.110@tcp100 [15289.887224] LNet: Removed LNI 192.168.202.110@tcp101 [15289.892112] LNetError: 467950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15289.909719] LNetError: 467950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15289.926710] LNet: Removed LNI 192.168.202.110@tcp102 [15290.975291] LNet: Removed LNI 192.168.202.110@tcp103 [15292.001548] LNet: Removed LNI 192.168.202.110@tcp104 [15292.023215] LNetError: 467950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15292.034475] LNetError: 467950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15292.041914] LNet: Removed LNI 192.168.202.110@tcp105 [15293.089692] LNet: Removed LNI 192.168.202.110@tcp106 [15294.111451] LNet: Removed LNI 192.168.202.110@tcp107 [15295.135763] LNet: Removed LNI 192.168.202.110@tcp108 [15296.159271] LNet: Removed LNI 192.168.202.110@tcp109 [15296.161328] LNetError: 467950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15296.174821] LNetError: 467950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15296.182941] LNet: Removed LNI 192.168.202.110@tcp110 [15297.248293] LNet: Removed LNI 192.168.202.110@tcp111 [15298.271204] LNet: Removed LNI 192.168.202.110@tcp112 [15299.295148] LNet: Removed LNI 192.168.202.110@tcp113 [15300.319492] LNet: Removed LNI 192.168.202.110@tcp114 [15301.343697] LNet: Removed LNI 192.168.202.110@tcp115 [15302.367172] LNet: Removed LNI 192.168.202.110@tcp116 [15303.455261] LNet: Removed LNI 192.168.202.110@tcp117 [15304.481162] LNet: Removed LNI 192.168.202.110@tcp118 [15304.499627] LNetError: 467950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15304.536092] LNetError: 467950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15304.549764] LNet: Removed LNI 192.168.202.110@tcp119 [15305.631188] LNet: Removed LNI 192.168.202.110@tcp120 [15306.719205] LNet: Removed LNI 192.168.202.110@tcp121 [15307.782082] LNet: Removed LNI 192.168.202.110@tcp122 [15307.789661] LNet: Removed LNI 192.168.202.110@tcp123 [15308.831266] LNet: Removed LNI 192.168.202.110@tcp124 [15309.860868] LNet: Removed LNI 192.168.202.110@tcp125 [15310.880847] LNet: Removed LNI 192.168.202.110@tcp126 [15311.903153] LNet: Removed LNI 192.168.202.110@tcp127 [15312.927293] LNet: Removed LNI 192.168.202.110@tcp128 [15313.951563] LNet: Removed LNI 192.168.202.110@tcp129 [15313.958117] LNet: Removed LNI 192.168.202.110@tcp130 [15313.967950] LNet: Removed LNI 192.168.202.110@tcp131 [15313.974695] LNet: Removed LNI 192.168.202.110@tcp132 [15313.979589] LNet: Removed LNI 192.168.202.110@tcp133 [15313.986946] LNet: Removed LNI 192.168.202.110@tcp134 [15313.999176] LNet: Removed LNI 192.168.202.110@tcp135 [15314.855522] Key type .llcrypt unregistered [15314.857907] Key type ._llcrypt unregistered [15343.114796] Key type ._llcrypt registered [15343.117238] Key type .llcrypt registered [15343.333789] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [15361.910080] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [15364.305520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15364.381112] alg: No test for adler32 (adler32-zlib) [15365.556771] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [15365.987744] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [15367.815273] Key type lgssc registered [15369.320297] Lustre: Echo OBD driver; http://www.lustre.org/ [15409.206557] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:09:36 (1788858576) [15430.126742] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [15433.227524] Key type lgssc unregistered [15433.572658] LNet: 472038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15433.584042] LNetError: 472038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15433.598969] LNet: Removed LNI 192.168.202.110@tcp [15434.760293] Key type .llcrypt unregistered [15434.762447] Key type ._llcrypt unregistered [15456.435506] Key type ._llcrypt registered [15456.438237] Key type .llcrypt registered [15456.552814] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [15458.231102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15458.265720] alg: No test for adler32 (adler32-zlib) [15459.408357] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [15459.712919] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [15461.368611] Key type lgssc registered [15462.759310] Lustre: Echo OBD driver; http://www.lustre.org/ [15473.322207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15473.351725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15474.635623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15474.674789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15474.769424] Lustre: lustre-MDT0000: new disk, initializing [15474.945920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15474.969839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15479.807862] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15492.439208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15492.591298] Lustre: 473248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15492.638443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15492.641840] Lustre: Skipped 1 previous similar message [15492.716548] Lustre: lustre-MDT0001: new disk, initializing [15492.849434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15492.908025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15492.932099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15498.249643] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15508.747307] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15515.580472] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15522.679373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15523.067804] Lustre: lustre-OST0000: new disk, initializing [15523.071476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15523.077556] Lustre: 475216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15523.184933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15531.764806] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15532.259483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15532.276569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15532.474741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15543.709391] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15549.308675] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing lnet_if_list [15556.358224] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15556.394430] LNet: Added LNI 192.168.202.110@tcp10 [8/256/0/180] [15562.857925] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15562.894697] LNet: Added LNI 192.168.202.110@tcp11 [8/256/0/180] [15567.805758] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15567.928906] LNet: Added LNI 192.168.202.110@tcp12 [8/256/0/180] [15572.600802] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15572.619882] LNet: Added LNI 192.168.202.110@tcp13 [8/256/0/180] [15578.999067] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15579.026686] LNet: Added LNI 192.168.202.110@tcp14 [8/256/0/180] [15583.754742] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15583.771821] LNet: Added LNI 192.168.202.110@tcp15 [8/256/0/180] [15591.413333] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15591.470599] LNet: Added LNI 192.168.202.110@tcp16 [8/256/0/180] [15598.676863] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15598.723480] LNet: Added LNI 192.168.202.110@tcp17 [8/256/0/180] [15604.603911] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15604.643967] LNet: Added LNI 192.168.202.110@tcp18 [8/256/0/180] [15610.721864] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15610.756334] LNet: Added LNI 192.168.202.110@tcp19 [8/256/0/180] [15617.173076] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15617.230993] LNet: Added LNI 192.168.202.110@tcp20 [8/256/0/180] [15624.441692] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15624.478224] LNet: Added LNI 192.168.202.110@tcp21 [8/256/0/180] [15631.114951] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15631.158420] LNet: Added LNI 192.168.202.110@tcp22 [8/256/0/180] [15638.569604] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15638.624540] LNet: Added LNI 192.168.202.110@tcp23 [8/256/0/180] [15645.482746] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15645.513934] LNet: Added LNI 192.168.202.110@tcp24 [8/256/0/180] [15652.931331] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15652.968957] LNet: Added LNI 192.168.202.110@tcp25 [8/256/0/180] [15660.015134] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15660.105442] LNet: Added LNI 192.168.202.110@tcp26 [8/256/0/180] [15667.506561] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15667.546681] LNet: Added LNI 192.168.202.110@tcp27 [8/256/0/180] [15674.445253] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15674.469339] LNet: Added LNI 192.168.202.110@tcp28 [8/256/0/180] [15680.965540] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15681.024945] LNet: Added LNI 192.168.202.110@tcp29 [8/256/0/180] [15688.840625] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15688.861340] LNet: Added LNI 192.168.202.110@tcp30 [8/256/0/180] [15696.121378] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15696.146645] LNet: Added LNI 192.168.202.110@tcp31 [8/256/0/180] [15703.336366] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15703.371975] LNet: Added LNI 192.168.202.110@tcp32 [8/256/0/180] [15711.644245] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15711.688502] LNet: Added LNI 192.168.202.110@tcp33 [8/256/0/180] [15719.446271] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15719.487112] LNet: Added LNI 192.168.202.110@tcp34 [8/256/0/180] [15726.625401] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15726.671420] LNet: Added LNI 192.168.202.110@tcp35 [8/256/0/180] [15734.054550] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15734.084687] LNet: Added LNI 192.168.202.110@tcp36 [8/256/0/180] [15742.757787] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15742.840425] LNet: Added LNI 192.168.202.110@tcp37 [8/256/0/180] [15750.756912] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15750.802526] LNet: Added LNI 192.168.202.110@tcp38 [8/256/0/180] [15760.719708] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15760.786211] LNet: Added LNI 192.168.202.110@tcp39 [8/256/0/180] [15770.371540] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15770.392871] LNet: Added LNI 192.168.202.110@tcp40 [8/256/0/180] [15780.287610] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15780.345529] LNet: Added LNI 192.168.202.110@tcp41 [8/256/0/180] [15790.355666] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15790.408106] LNet: Added LNI 192.168.202.110@tcp42 [8/256/0/180] [15801.169749] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15801.207236] LNet: Added LNI 192.168.202.110@tcp43 [8/256/0/180] [15811.119496] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15811.142279] LNet: Added LNI 192.168.202.110@tcp44 [8/256/0/180] [15820.865630] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15820.922161] LNet: Added LNI 192.168.202.110@tcp45 [8/256/0/180] [15830.088788] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15830.135550] LNet: Added LNI 192.168.202.110@tcp46 [8/256/0/180] [15839.753739] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15839.787517] LNet: Added LNI 192.168.202.110@tcp47 [8/256/0/180] [15850.757261] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15850.788686] LNet: Added LNI 192.168.202.110@tcp48 [8/256/0/180] [15861.766743] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15861.808994] LNet: Added LNI 192.168.202.110@tcp49 [8/256/0/180] [15874.050593] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15874.098660] LNet: Added LNI 192.168.202.110@tcp50 [8/256/0/180] [15896.034115] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15896.064786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15896.076938] Lustre: Skipped 2 previous similar messages [15901.298691] Lustre: server umount lustre-MDT0000 complete [15905.724987] LustreError: 473242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859075 with bad export cookie 10825919091808095923 [15905.729896] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15905.736281] LustreError: 473242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15906.130918] Lustre: server umount lustre-MDT0001 complete [15922.399847] Lustre: 472522:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788859076/real 1788859076] req@0000000002f2e9c8 x1875754225955456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788859092 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15922.432740] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15922.462872] Lustre: Skipped 2 previous similar messages [15926.754178] Lustre: 472519:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788859080/real 1788859080] req@00000000c2409e24 x1875754225955584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788859096 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15927.842232] Lustre: 472519:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788859081/real 1788859081] req@00000000efc53422 x1875754225955712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788859097 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15929.648602] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [15931.874992] Lustre: 472521:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788859085/real 1788859085] req@0000000068d6a093 x1875754225955968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788859101 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15937.056885] Lustre: 472521:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788859090/real 1788859090] req@00000000d56b7b59 x1875754225956096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788859106 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15941.512825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15951.213279] LustreError: 472518:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ce4fbe9b x1875754225956864/t0(0) o250->MGC192.168.202.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15951.626890] LustreError: 475210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15951.653575] LustreError: 475210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15951.698620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15953.702441] LustreError: 475211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15956.703981] LustreError: 475210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15957.866240] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15962.081373] LustreError: 475211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15967.211298] LustreError: 475210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15967.229123] LustreError: 475210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15968.532503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15969.044704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15974.394514] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15974.528656] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15976.438756] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15986.803291] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15995.049964] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16020.451996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16020.469286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16020.494186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16022.505656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16022.512718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16025.383265] Lustre: server umount lustre-OST0000 complete [16030.692560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16030.699851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16030.722674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16030.731395] Lustre: Skipped 1 previous similar message [16036.512151] Lustre: server umount lustre-MDT0000 complete [16037.856612] LustreError: 499470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16037.865059] LustreError: 499470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16040.092896] LustreError: 476427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859209 with bad export cookie 10825919091808098639 [16040.093345] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16040.103658] LustreError: 476427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16040.507594] Lustre: server umount lustre-MDT0001 complete [16053.609445] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [16057.098719] Key type lgssc unregistered [16057.819384] LNet: 502275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16057.838715] LNetError: 502275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16058.847288] LNet: Removed LNI 192.168.202.110@tcp [16058.849371] LNetError: 502275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16058.855704] LNet: Removed LNI 192.168.202.110@tcp10 [16058.859147] LNet: Removed LNI 192.168.202.110@tcp11 [16058.865446] LNet: Removed LNI 192.168.202.110@tcp12 [16058.870866] LNet: Removed LNI 192.168.202.110@tcp13 [16058.877847] LNet: Removed LNI 192.168.202.110@tcp14 [16058.884840] LNet: Removed LNI 192.168.202.110@tcp15 [16058.890099] LNet: Removed LNI 192.168.202.110@tcp16 [16058.900301] LNet: Removed LNI 192.168.202.110@tcp17 [16058.912967] LNet: Removed LNI 192.168.202.110@tcp18 [16058.928160] LNet: Removed LNI 192.168.202.110@tcp19 [16058.935953] LNet: Removed LNI 192.168.202.110@tcp20 [16058.940406] LNet: Removed LNI 192.168.202.110@tcp21 [16058.945539] LNet: Removed LNI 192.168.202.110@tcp22 [16058.949964] LNet: Removed LNI 192.168.202.110@tcp23 [16058.953644] LNet: Removed LNI 192.168.202.110@tcp24 [16058.964312] LNet: Removed LNI 192.168.202.110@tcp25 [16058.966440] LNet: Removed LNI 192.168.202.110@tcp26 [16058.979650] LNet: Removed LNI 192.168.202.110@tcp27 [16058.990344] LNet: Removed LNI 192.168.202.110@tcp28 [16058.997601] LNet: Removed LNI 192.168.202.110@tcp29 [16059.007845] LNet: Removed LNI 192.168.202.110@tcp30 [16059.027782] LNet: Removed LNI 192.168.202.110@tcp31 [16059.038391] LNet: Removed LNI 192.168.202.110@tcp32 [16059.040534] LNet: Removed LNI 192.168.202.110@tcp33 [16059.064729] LNet: Removed LNI 192.168.202.110@tcp34 [16059.091702] LNet: Removed LNI 192.168.202.110@tcp35 [16059.097902] LNet: Removed LNI 192.168.202.110@tcp36 [16059.115829] LNet: Removed LNI 192.168.202.110@tcp37 [16059.117790] LNet: Removed LNI 192.168.202.110@tcp38 [16059.132444] LNet: Removed LNI 192.168.202.110@tcp39 [16059.134942] LNet: Removed LNI 192.168.202.110@tcp40 [16059.150542] LNet: Removed LNI 192.168.202.110@tcp41 [16059.156981] LNet: Removed LNI 192.168.202.110@tcp42 [16059.158930] LNet: Removed LNI 192.168.202.110@tcp43 [16059.164266] LNet: Removed LNI 192.168.202.110@tcp44 [16059.167332] LNet: Removed LNI 192.168.202.110@tcp45 [16059.169915] LNet: Removed LNI 192.168.202.110@tcp46 [16059.172570] LNet: Removed LNI 192.168.202.110@tcp47 [16059.174814] LNet: Removed LNI 192.168.202.110@tcp48 [16059.177725] LNet: Removed LNI 192.168.202.110@tcp49 [16059.188901] LNet: Removed LNI 192.168.202.110@tcp50 [16060.562517] Key type .llcrypt unregistered [16060.566422] Key type ._llcrypt unregistered [16087.055924] Key type ._llcrypt registered [16087.057863] Key type .llcrypt registered [16087.195857] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [16108.593568] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [16110.106577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16110.266594] alg: No test for adler32 (adler32-zlib) [16111.610955] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [16111.898891] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [16113.639281] Key type lgssc registered [16115.718324] Lustre: Echo OBD driver; http://www.lustre.org/ [16165.514962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16165.554364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16166.910615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16166.948373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16167.040805] Lustre: lustre-MDT0000: new disk, initializing [16167.188584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16167.254281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16172.685986] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16186.648140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16186.896340] Lustre: 505474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16186.985191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16186.996966] Lustre: Skipped 1 previous similar message [16187.091267] Lustre: lustre-MDT0001: new disk, initializing [16187.204222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16187.246829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16187.265814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16192.800213] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16203.242208] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16209.683786] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16219.786892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16220.081204] Lustre: *** cfs_fail_loc=90f, val=0*** [16220.252544] Lustre: lustre-OST0000: new disk, initializing [16220.259984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16220.268387] Lustre: 507494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16220.446588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16227.039548] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16228.903346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16228.916359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16228.990580] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16238.092453] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16249.312606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16249.323665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16249.352821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16249.828698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16249.843696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16249.871491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16253.785374] Lustre: server umount lustre-OST0000 complete [16259.040255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16259.044729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16259.060888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16264.815694] Lustre: server umount lustre-MDT0000 complete [16268.708310] LustreError: 505466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859438 with bad export cookie 18364109438903450619 [16268.711016] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16268.725719] LustreError: 505466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16269.079288] Lustre: server umount lustre-MDT0001 complete [16286.717313] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [16290.295623] Key type lgssc unregistered [16290.795982] LNet: 509399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16290.807776] LNetError: 509399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16290.832885] LNet: Removed LNI 192.168.202.110@tcp [16291.939428] Key type .llcrypt unregistered [16291.945563] Key type ._llcrypt unregistered [16317.424661] Key type ._llcrypt registered [16317.426718] Key type .llcrypt registered [16317.554267] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [16318.919051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16318.967755] alg: No test for adler32 (adler32-zlib) [16320.219197] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [16320.574387] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [16322.375258] Key type lgssc registered [16324.083759] Lustre: Echo OBD driver; http://www.lustre.org/ [16333.809347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16333.828926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16335.362995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16340.771224] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16345.569971] LustreError: 510667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16345.582663] LustreError: 510667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16350.689268] LustreError: 510666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16350.826639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16351.097658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16356.313670] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16365.473303] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16373.203056] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16382.189897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16382.765647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16390.995370] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16401.337793] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16417.627145] loop0: detected capacity change from 0 to 204800000 [16417.883200] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16417.909468] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16417.933332] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16427.070907] loop0: detected capacity change from 0 to 204800000 [16427.369744] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16427.404511] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16427.420825] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16434.109869] loop0: detected capacity change from 0 to 204800000 [16441.304217] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16441.543642] Lustre: fs15246-OST0000: new disk, initializing [16441.547911] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16441.555968] Lustre: 514264:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16441.687902] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16449.455901] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16461.032687] loop1: detected capacity change from 0 to 204800000 [16467.386105] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16467.542836] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16467.597428] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16467.677109] Lustre: fs15246-MDT0000: new disk, initializing [16467.863682] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16467.894608] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16468.024539] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16469.089240] Lustre: 515387:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000084fcb79 x1875755128641024/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788859654 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16473.880721] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16478.189043] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16478.198256] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16478.240029] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16505.328112] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16505.354177] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16508.520132] Lustre: server umount fs15246-MDT0000 complete [16510.437070] LustreError: 510662:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16515.558951] LustreError: 510662:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16519.399565] Lustre: server umount fs15246-OST0000 complete [16536.035905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16536.052371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16536.074034] Lustre: Skipped 1 previous similar message [16536.091221] Lustre: Skipped 1 previous similar message [16541.156858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16541.165284] Lustre: Skipped 1 previous similar message [16542.024174] Lustre: server umount lustre-OST0000 complete [16550.885936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16550.889499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16550.910736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16553.527951] Lustre: server umount lustre-MDT0000 complete [16556.518764] LustreError: 512260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16556.548817] LustreError: 512260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16558.133188] LustreError: 510647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859727 with bad export cookie 8744324286144694813 [16558.140183] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16558.155586] LustreError: 510647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16558.643946] Lustre: server umount lustre-MDT0001 complete [16576.185310] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [16579.875541] Key type lgssc unregistered [16580.226772] LNet: 518504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16580.234915] LNetError: 518504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16580.250618] LNet: Removed LNI 192.168.202.110@tcp [16581.457715] Key type .llcrypt unregistered [16581.460263] Key type ._llcrypt unregistered [16601.783751] Key type ._llcrypt registered [16601.785448] Key type .llcrypt registered [16601.885568] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [16603.345735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16603.453080] alg: No test for adler32 (adler32-zlib) [16604.842182] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [16605.222589] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [16607.071206] Key type lgssc registered [16608.983855] Lustre: Echo OBD driver; http://www.lustre.org/ [16662.764913] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [16675.527742] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [16724.205314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16724.232453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16725.584383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16725.619429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16725.681356] Lustre: lustre-MDT0000: new disk, initializing [16725.753024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16725.772682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16730.399324] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16743.654804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16743.714992] Lustre: 522971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16743.761424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16743.764466] Lustre: Skipped 1 previous similar message [16743.847645] Lustre: lustre-MDT0001: new disk, initializing [16743.925648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16743.951664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16743.968270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16748.986242] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16760.310663] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16767.979282] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16777.171308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16777.771853] Lustre: lustre-OST0000: new disk, initializing [16777.777838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16777.792363] Lustre: 524945:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16777.982694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16784.506556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16784.524701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16784.728423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16785.347434] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16797.220124] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16804.123165] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16804.352928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16812.700818] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16813.131921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16820.711493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16820.724989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16820.734352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16820.743106] Lustre: Skipped 1 previous similar message [16822.183522] Lustre: server umount lustre-OST0000 complete [16830.945838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16830.951420] LustreError: Skipped 1 previous similar message [16830.957210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16830.961237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16830.990421] Lustre: Skipped 2 previous similar messages [16832.607080] Lustre: server umount lustre-MDT0000 complete [16836.629098] LustreError: 522965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860006 with bad export cookie 15411116642289466608 [16836.630235] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16836.638844] LustreError: 522965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16837.048585] Lustre: server umount lustre-MDT0001 complete [16866.916761] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [16878.361890] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [16916.210772] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16553 sec ========= 05:34:43 (1788860083) [16918.282588] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:34:45 (1788860085) === [16922.757399] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:34:49 (1788860089) === [16969.008243] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [16972.495389] Key type lgssc unregistered [16972.902695] LNet: 531593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16972.917256] LNetError: 531593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16972.949762] LNet: Removed LNI 192.168.202.110@tcp [16973.893482] Key type .llcrypt unregistered [16973.895314] Key type ._llcrypt unregistered