[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 483657517 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003232] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.014045] *** VALIDATE selinux *** [ 0.022563] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027250] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029129] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030109] *** VALIDATE tmpfs *** [ 0.031465] *** VALIDATE proc *** [ 0.033004] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.036216] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.042950] debug: unmapping init [mem 0xffffffffb7059000-0xffffffffb7060fff] [ 0.044176] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045732] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053314] rcu: Hierarchical SRCU implementation. [ 0.055417] smp: Bringing up secondary CPUs ... [ 0.056567] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.060585] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.153211] node 0 deferred pages initialised in 86ms [ 0.156099] devtmpfs: initialized [ 0.158282] x86/mm: Memory block size: 128MB [ 0.161735] gcov: version magic: 0x41383552 [ 0.164244] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.168120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.170315] pinctrl core: initialized pinctrl subsystem [ 0.172260] [ 0.172552] ************************************************************* [ 0.175014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177014] ** ** [ 0.180016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.182011] ** ** [ 0.184014] ** This means that this kernel is built to expose internal ** [ 0.186014] ** IOMMU data structures, which may compromise security on ** [ 0.188013] ** your system. ** [ 0.191016] ** ** [ 0.193012] ** If you see this message and you are not debugging the ** [ 0.195013] ** kernel, report this immediately to your vendor! ** [ 0.198015] ** ** [ 0.201014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.203012] ************************************************************* [ 0.206810] NET: Registered protocol family 16 [ 0.208432] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.211060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.213059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.217172] cpuidle: using governor menu [ 0.218624] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.221989] PCI: Using configuration type 1 for base access [ 0.224155] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.236093] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.239084] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.243041] cryptd: max_cpu_qlen set to 1000 [ 0.247327] ACPI: Added _OSI(Module Device) [ 0.249017] ACPI: Added _OSI(Processor Device) [ 0.250015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.251014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.255272] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.262519] ACPI: Interpreter enabled [ 0.264054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.265013] ACPI: Using IOAPIC for interrupt routing [ 0.267171] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.270440] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.280103] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.283065] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.286031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.290107] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.295579] acpiphp: Slot [2] registered [ 0.298221] acpiphp: Slot [5] registered [ 0.299209] acpiphp: Slot [6] registered [ 0.301213] acpiphp: Slot [7] registered [ 0.303212] acpiphp: Slot [8] registered [ 0.304217] acpiphp: Slot [9] registered [ 0.306201] acpiphp: Slot [10] registered [ 0.308128] acpiphp: Slot [3] registered [ 0.312170] acpiphp: Slot [4] registered [ 0.313125] acpiphp: Slot [11] registered [ 0.315136] acpiphp: Slot [12] registered [ 0.317180] acpiphp: Slot [13] registered [ 0.319142] acpiphp: Slot [14] registered [ 0.322273] acpiphp: Slot [15] registered [ 0.324272] acpiphp: Slot [16] registered [ 0.326112] acpiphp: Slot [17] registered [ 0.327116] acpiphp: Slot [18] registered [ 0.329111] acpiphp: Slot [19] registered [ 0.331210] acpiphp: Slot [20] registered [ 0.332146] acpiphp: Slot [21] registered [ 0.334150] acpiphp: Slot [22] registered [ 0.336146] acpiphp: Slot [23] registered [ 0.337186] acpiphp: Slot [24] registered [ 0.339143] acpiphp: Slot [25] registered [ 0.341204] acpiphp: Slot [26] registered [ 0.342155] acpiphp: Slot [27] registered [ 0.344155] acpiphp: Slot [28] registered [ 0.346147] acpiphp: Slot [29] registered [ 0.348174] acpiphp: Slot [30] registered [ 0.350143] acpiphp: Slot [31] registered [ 0.351085] PCI host bridge to bus 0000:00 [ 0.353032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.356034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.359033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.362035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.365037] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.369037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.371243] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.375186] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.379642] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.391958] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.396989] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.399025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.402024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.405023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.407523] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.411175] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.414067] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.417912] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.423025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.437025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.445021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.450614] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.457018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.463014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.478018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.489069] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.498024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.505022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.527018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.537873] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.547020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.554020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.571017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.580910] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.591017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.598017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.614017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.624048] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.629019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.638024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.660024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.673000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.681020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.689017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.707025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.716960] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.719486] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.722438] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.726677] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.729414] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.735344] iommu: Default domain type: Passthrough [ 0.736414] SCSI subsystem initialized [ 0.738118] ACPI: bus type USB registered [ 0.739120] usbcore: registered new interface driver usbfs [ 0.741075] usbcore: registered new interface driver hub [ 0.744084] usbcore: registered new device driver usb [ 0.746305] pps_core: LinuxPPS API ver. 1 registered [ 0.748011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.751081] PTP clock support registered [ 0.753147] EDAC MC: Ver: 3.0.0 [ 0.755105] PCI: Using ACPI for IRQ routing [ 0.758051] NetLabel: Initializing [ 0.759011] NetLabel: domain hash size = 128 [ 0.761009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.763080] NetLabel: unlabeled traffic allowed by default [ 0.766139] vgaarb: loaded [ 0.767250] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.768011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.774374] clocksource: Switched to clocksource kvm-clock [ 0.883715] VFS: Disk quotas dquot_6.6.0 [ 0.884870] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.886916] *** VALIDATE ramfs *** [ 0.887922] *** VALIDATE hugetlbfs *** [ 0.889621] pnp: PnP ACPI init [ 0.892464] pnp: PnP ACPI: found 6 devices [ 0.912572] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.916194] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.918407] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.920014] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.922150] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.923950] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.926622] NET: Registered protocol family 2 [ 0.929105] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.933935] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.937586] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.943484] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.946937] TCP: Hash tables configured (established 65536 bind 65536) [ 0.949854] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.953307] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.956210] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.959092] NET: Registered protocol family 1 [ 0.961670] RPC: Registered named UNIX socket transport module. [ 0.963418] RPC: Registered udp transport module. [ 0.964729] RPC: Registered tcp transport module. [ 0.966475] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.968715] NET: Registered protocol family 44 [ 0.970345] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.971881] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.973602] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.975988] PCI: CLS 0 bytes, default 64 [ 0.978410] Unpacking initramfs... [ 2.332465] debug: unmapping init [mem 0xffff9efcfcc54000-0xffff9efcfffbffff] [ 2.335981] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.338608] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.341566] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.820904] Initialise system trusted keyrings [ 2.822355] Key type blacklist registered [ 2.824337] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.832872] zbud: loaded [ 2.835796] *** VALIDATE nfs *** [ 2.836563] *** VALIDATE nfs4 *** [ 2.839072] pstore: using deflate compression [ 2.841624] Platform Keyring initialized [ 2.931384] NET: Registered protocol family 38 [ 2.932938] Key type asymmetric registered [ 2.934085] Asymmetric key parser 'x509' registered [ 2.936190] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.939279] io scheduler mq-deadline registered [ 2.940873] io scheduler kyber registered [ 2.942547] io scheduler bfq registered [ 2.944592] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.947784] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.950685] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.953623] ACPI: Power Button [PWRF] [ 2.958870] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.965610] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.985427] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.996705] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.015578] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.041546] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.070930] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.076615] Non-volatile memory driver v1.3 [ 3.078452] Linux agpgart interface v0.103 [ 3.111569] virtio_blk virtio1: [vda] 149448 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.115487] vda: detected capacity change from 0 to 76517376 [ 3.130064] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.133029] vdb: detected capacity change from 0 to 1073741824 [ 3.149964] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.153021] vdc: detected capacity change from 0 to 2621440000 [ 3.168478] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.171766] vdd: detected capacity change from 0 to 2621440000 [ 3.189274] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.192509] vde: detected capacity change from 0 to 4294967296 [ 3.208584] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.212361] vdf: detected capacity change from 0 to 4294967296 [ 3.226456] libphy: Fixed MDIO Bus: probed [ 3.232048] usbcore: registered new interface driver usbserial_generic [ 3.235554] usbserial: USB Serial support registered for generic [ 3.238378] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.245591] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.247362] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.249784] mousedev: PS/2 mouse device common for all mice [ 3.252830] rtc_cmos 00:05: RTC can wake from S4 [ 3.253888] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.257789] rtc_cmos 00:05: registered as rtc0 [ 3.261687] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.264671] intel_pstate: CPU model not supported [ 3.267491] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.271629] hid: raw HID events driver (C) Jiri Kosina [ 3.273766] usbcore: registered new interface driver usbhid [ 3.275709] usbhid: USB HID core driver [ 3.275894] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.277385] drop_monitor: Initializing network drop monitor service [ 3.283391] Initializing XFRM netlink socket [ 3.286048] NET: Registered protocol family 10 [ 3.288440] Segment Routing with IPv6 [ 3.290217] NET: Registered protocol family 17 [ 3.292474] mpls_gso: MPLS GSO support [ 3.298113] RAS: Correctable Errors collector initialized. [ 3.300723] AVX version of gcm_enc/dec engaged. [ 3.302562] AES CTR mode by8 optimization enabled [ 3.384753] sched_clock: Marking stable (3384731593, 0)->(4279098979, -894367386) [ 3.388261] registered taskstats version 1 [ 3.389677] Loading compiled-in X.509 certificates [ 3.391291] zswap: loaded using pool lzo/zbud [ 3.414147] Key type big_key registered [ 3.424915] Key type encrypted registered [ 3.426626] ima: No TPM chip found, activating TPM-bypass! [ 3.428812] ima: Allocated hash algorithm: sha1 [ 3.431327] ima: No architecture policies found [ 3.433303] evm: Initialising EVM extended attributes: [ 3.435551] evm: security.selinux [ 3.437062] evm: security.ima [ 3.438374] evm: security.capability [ 3.439815] evm: HMAC attrs: 0x1 [ 3.442331] rtc_cmos 00:05: setting system clock to 2026-08-22 08:01:19 UTC (1787385679) [ 3.449063] debug: unmapping init [mem 0xffffffffb8003000-0xffffffffb81fffff] [ 3.452483] debug: unmapping init [mem 0xffffffffb6d82000-0xffffffffb7058fff] [ 3.464176] Write protecting the kernel read-only data: 28672k [ 3.469072] debug: unmapping init [mem 0xffffffffb5403000-0xffffffffb55fffff] [ 3.471775] debug: unmapping init [mem 0xffffffffb5d14000-0xffffffffb5dfffff] [ 3.505976] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.514562] systemd[1]: Detected virtualization kvm. [ 3.516083] systemd[1]: Detected architecture x86-64. [ 3.517912] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.543092] systemd[1]: No hostname configured. [ 3.544682] systemd[1]: Set hostname to . [ 3.546544] random: systemd: uninitialized urandom read (16 bytes read) [ 3.548583] systemd[1]: Initializing machine ID from random generator. [ 3.680989] random: systemd: uninitialized urandom read (16 bytes read) [ 3.684127] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.704931] random: systemd: uninitialized urandom read (16 bytes read) [ 3.709376] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.714693] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.345304] device-mapper: uevent: version 1.0.3 [ 4.347536] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [[ 4.893624] random: fast init done  OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.083906] virtio_net virtio0 ens2: renamed from eth0 [ 5.146348] scsi host0: ata_piix [ 5.157444] scsi host1: ata_piix [ 5.159146] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.161096] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.951712] random: crng init done [ 9.959939] random: 7 urandom warning(s) missed due to ratelimiting [ 10.047211] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.344456] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.491566] printk: systemd: 25 output lines suppressed due to ratelimiting [ 14.201900] SELinux: Disabled at runtime. [ 14.310091] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.322915] systemd[1]: Detected virtualization kvm. [ 14.325586] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.394463] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.397500] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.406634] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.417065] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.425192] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.438907] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.449384] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK [[ 15.791365] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 16.421545] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.227019] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.261562] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.464049] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.520854] EDAC sbridge: Ver: 1.1.2 [ 21.689558] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (6s / no limit)[ 22.557547] NFS: Registering the id_resolver key type [ 22.561764] Key type id_resolver registered [ 22.565979] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg607-server login: [ 40.174005] hrtimer: interrupt took 11995981 ns [ 87.199732] libcfs: loading out-of-tree module taints kernel. [ 87.292326] Key type ._llcrypt registered [ 87.294730] Key type .llcrypt registered [ 87.435061] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 105.425967] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 107.400968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 107.417099] alg: No test for adler32 (adler32-zlib) [ 109.025132] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 110.257211] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 112.192162] Key type lgssc registered [ 114.248376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 132.681296] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.765509] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 185.569566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 185.649857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.118563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 187.144268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 187.283383] Lustre: lustre-MDT0000: new disk, initializing [ 187.478402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.515332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 192.382154] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.488240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.627273] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 206.659293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 206.662991] Lustre: Skipped 1 previous similar message [ 206.756306] Lustre: lustre-MDT0001: new disk, initializing [ 206.824086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.851513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 206.862710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 212.528584] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.963155] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 228.111965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.371972] Lustre: lustre-OST0000: new disk, initializing [ 228.375460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 228.383735] Lustre: 8448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 228.493483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.345531] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.601450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 236.619177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 236.771579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 250.978612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.151519] Lustre: lustre-OST0001: new disk, initializing [ 251.154593] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 251.161660] Lustre: 9521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 251.240138] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.687435] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.177849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 261.190739] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 261.274409] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 269.487147] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 274.323671] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 280.701181] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing check_logdir /tmp/testlogs/ [ 286.646318] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing yml_node [ 290.787483] Lustre: DEBUG MARKER: Client: 2.17.54.112 [ 293.678961] Lustre: DEBUG MARKER: MDS: 2.17.54.112 [ 297.078381] Lustre: DEBUG MARKER: OSS: 2.17.54.112 [ 299.263835] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 04:06:13 EDT 2026 [ 317.645749] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 319.173231] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.402253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.403726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.417699] Lustre: Skipped 1 previous similar message [ 341.399715] Lustre: server umount lustre-MDT0000 complete [ 345.072314] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386021 with bad export cookie 2582940134293053882 [ 345.078578] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 345.084921] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 348.641319] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.641786] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.643308] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.643314] Lustre: Skipped 3 previous similar messages [ 348.677023] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 348.684762] Lustre: Skipped 3 previous similar messages [ 350.688712] LustreError: 7723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 350.696537] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 350.717188] LustreError: 7723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 350.723976] Lustre: Skipped 2 previous similar messages [ 355.812242] LustreError: 12465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 355.812421] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 355.821357] LustreError: 12465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 355.837352] Lustre: Skipped 1 previous similar message [ 359.393226] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 359.617198] Lustre: server umount lustre-MDT0001 complete [ 363.983416] Lustre: server umount lustre-OST0000 complete [ 368.237445] Lustre: server umount lustre-OST0001 complete [ 375.234739] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 384.782551] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 425.039719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.281552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 425.307572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 425.394505] Lustre: lustre-MDT0000: new disk, initializing [ 425.497660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.532760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 430.104504] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 440.234307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.371656] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 440.468906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 440.471904] Lustre: Skipped 1 previous similar message [ 440.594890] Lustre: lustre-MDT0001: new disk, initializing [ 440.690853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 440.725889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 440.734114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 444.987432] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 453.996428] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 459.727982] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 466.223031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 466.470255] Lustre: lustre-OST0000: new disk, initializing [ 466.473262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 466.479222] Lustre: 17043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 466.597055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.724266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 467.737495] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 467.836086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 472.168883] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 480.990463] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 485.591200] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 485.763770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 489.976241] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 490.138668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 493.548260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 493.551887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.560239] LustreError: Skipped 1 previous similar message [ 493.582942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.785285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.806817] Lustre: Skipped 4 previous similar messages [ 506.347936] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 506.643604] Lustre: server umount lustre-OST0000 complete [ 512.483391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 512.497340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.520768] Lustre: Skipped 1 previous similar message [ 524.257311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 524.269628] Lustre: Skipped 6 previous similar messages [ 525.280205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 525.558577] Lustre: server umount lustre-MDT0000 complete [ 529.315096] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386205 with bad export cookie 2582940134293055415 [ 529.316488] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.328333] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 529.688268] Lustre: server umount lustre-MDT0001 complete [ 533.634581] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:10:08 (1787386208) [ 536.288674] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 538.604924] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:10:12 (1787386212) [ 545.299756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.882499] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.951672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.213189] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 551.403723] LustreError: 19458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.677229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.717388] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 572.775885] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 579.038908] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 586.535436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 592.638536] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 600.759517] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 614.882903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.887119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 614.902109] Lustre: Skipped 1 previous similar message [ 614.903868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 617.974731] Lustre: server umount lustre-OST0000 complete [ 621.555604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 624.035189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 624.053524] Lustre: Skipped 2 previous similar messages [ 627.756570] Lustre: server umount lustre-MDT0000 complete [ 630.242083] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.269170] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 631.551769] LustreError: 19438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386307 with bad export cookie 2582940134293056297 [ 631.571239] LustreError: 19438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 631.571989] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 631.961133] Lustre: server umount lustre-MDT0001 complete [ 642.402266] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 645.296734] Key type lgssc unregistered [ 645.569501] LNet: 22990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.582742] LNetError: 22990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 646.632572] LNet: Removed LNI 192.168.206.107@tcp [ 647.806511] Key type .llcrypt unregistered [ 647.808980] Key type ._llcrypt unregistered [ 670.962620] Key type ._llcrypt registered [ 670.965372] Key type .llcrypt registered [ 671.089166] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 672.413075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 672.456135] alg: No test for adler32 (adler32-zlib) [ 673.559570] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 673.793545] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 675.448227] Key type lgssc registered [ 676.807687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 684.590292] loop: module loaded [ 684.734489] loop0: detected capacity change from 0 to 204800000 [ 684.955982] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.963459] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.190650] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 691.103579] loop0: detected capacity change from 0 to 204800000 [ 691.350241] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 691.365103] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 691.378592] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 697.046980] loop0: detected capacity change from 0 to 204800000 [ 703.400935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 703.436570] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.719135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 704.770469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 704.849333] Lustre: lustre-MDT0000: new disk, initializing [ 704.965350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.006789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 709.859542] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 719.550401] loop1: detected capacity change from 0 to 204800000 [ 724.587128] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 724.886493] Lustre: lustre-OST07c6: new disk, initializing [ 724.891882] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 724.896300] Lustre: Skipped 1 previous similar message [ 724.913272] Lustre: 26179:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 724.996538] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 731.067532] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 731.248963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 731.264052] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 731.307610] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 736.810411] Lustre: 25166:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 736.816569] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 736.824774] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 736.832303] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 736.839819] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 736.850731] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 753.240541] Lustre: 25166:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 753.253722] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 753.264844] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 753.271912] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 753.277852] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 753.283808] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 757.958341] Lustre: 25168:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 757.971467] Lustre: 25168:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 757.979771] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 757.988524] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.994814] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 758.000771] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.005396] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 758.010837] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.016423] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 758.022661] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.028634] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 758.033749] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.176664] Lustre: 25167:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 761.185398] Lustre: 25167:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 761.195448] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 761.202733] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.214559] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 761.223450] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.238444] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 761.248443] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.264225] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 761.278728] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.290276] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 761.298041] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.648267] Lustre: 25168:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 765.657919] Lustre: 25168:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 765.669581] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 765.679511] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.688630] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 765.696496] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.711385] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 765.726983] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.734921] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 765.743181] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.750678] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 765.756717] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.033480] Lustre: 26695:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 777.046333] Lustre: 26695:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 777.060823] Lustre: 26695:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 777.069204] Lustre: 26695:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.077639] Lustre: 26695:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 777.082853] Lustre: 26695:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.087449] Lustre: 26695:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 777.092620] Lustre: 26695:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.102985] Lustre: 26695:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 777.114342] Lustre: 26695:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.126815] Lustre: 26695:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.142600] Lustre: 26695:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 823.266491] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 823.272909] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 823.280946] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 828.395199] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 829.113733] Lustre: server umount lustre-OST07c6 complete [ 838.161373] Lustre: server umount lustre-MDT0000 complete [ 854.419630] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 857.054684] Key type lgssc unregistered [ 857.340333] LNet: 28735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.348113] LNetError: 28735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.360065] LNet: Removed LNI 192.168.206.107@tcp [ 858.268960] Key type .llcrypt unregistered [ 858.272463] Key type ._llcrypt unregistered [ 877.104768] Key type ._llcrypt registered [ 877.109644] Key type .llcrypt registered [ 877.250325] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 878.503417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 878.527924] alg: No test for adler32 (adler32-zlib) [ 879.667227] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 879.873421] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 881.528174] Key type lgssc registered [ 882.699282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 889.593936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 889.633032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 891.120440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 895.734908] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 901.616830] LustreError: 30007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 901.644958] LustreError: 30007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 904.521463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.974345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 910.056146] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 917.219373] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 922.892923] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 929.976581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 930.245797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.484989] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 936.445249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 942.563312] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 962.022718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 962.028768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.039841] LustreError: Skipped 1 previous similar message [ 962.064719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 965.101202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 965.111215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 965.119833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 966.554494] Lustre: server umount lustre-OST0000 complete [ 970.213385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.220675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 970.231371] Lustre: Skipped 1 previous similar message [ 970.246674] Lustre: Skipped 2 previous similar messages [ 975.329263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 975.341894] Lustre: Skipped 1 previous similar message [ 976.507366] Lustre: server umount lustre-MDT0000 complete [ 980.038197] LustreError: 29989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386656 with bad export cookie 17718475674230650204 [ 980.050844] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.052958] LustreError: 29989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 980.424841] Lustre: server umount lustre-MDT0001 complete [ 990.038976] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 992.416624] Key type lgssc unregistered [ 992.692297] LNet: 33741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.699165] LNetError: 33741:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.718687] LNet: Removed LNI 192.168.206.107@tcp [ 993.391539] Key type .llcrypt unregistered [ 993.393837] Key type ._llcrypt unregistered [ 1013.370418] Key type ._llcrypt registered [ 1013.373798] Key type .llcrypt registered [ 1013.488827] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1014.790544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1015.144014] alg: No test for adler32 (adler32-zlib) [ 1016.231700] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 1016.430758] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1018.072171] Key type lgssc registered [ 1019.224699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1027.265761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1027.282367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.990287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1034.325794] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1038.306073] LustreError: 35013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.333852] LustreError: 35013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1043.431680] LustreError: 35014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1043.817914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.288244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1049.509899] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1058.694439] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1065.847919] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1074.577202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1075.120515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1082.434853] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1090.970531] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1098.698768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1114.093579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1114.094394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.097193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1114.147487] Lustre: Skipped 1 previous similar message [ 1117.346481] Lustre: server umount lustre-OST0000 complete [ 1124.332092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.335868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1124.349978] Lustre: Skipped 1 previous similar message [ 1124.361702] Lustre: Skipped 2 previous similar messages [ 1127.949050] Lustre: server umount lustre-MDT0000 complete [ 1129.442612] LustreError: 35013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.465788] LustreError: 35013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1132.722699] LustreError: 34993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386808 with bad export cookie 7064452398995415134 [ 1132.738557] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1133.203452] Lustre: server umount lustre-MDT0001 complete [ 1146.411141] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1149.543990] Key type lgssc unregistered [ 1149.865741] LNet: 38576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.874157] LNetError: 38576:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1149.897907] LNet: Removed LNI 192.168.206.107@tcp [ 1150.785651] Key type .llcrypt unregistered [ 1150.787490] Key type ._llcrypt unregistered [ 1176.325599] Key type ._llcrypt registered [ 1176.328203] Key type .llcrypt registered [ 1176.423398] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1177.644043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1177.703552] alg: No test for adler32 (adler32-zlib) [ 1178.762914] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 1178.980967] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1180.648413] Key type lgssc registered [ 1181.864734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1190.103899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1190.130557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1191.543341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.077337] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1201.122515] LustreError: 39996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.141257] LustreError: 39996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1204.490771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1204.872594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1209.597355] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1216.731771] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1222.647939] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1230.202194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1230.627955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.392917] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1236.786947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1244.117844] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1254.374316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.379651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1254.394214] Lustre: Skipped 2 previous similar messages [ 1254.413172] Lustre: Skipped 2 previous similar messages [ 1256.956439] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.7@tcp (stopping) [ 1259.490237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1259.500047] Lustre: Skipped 2 previous similar messages [ 1260.178529] Lustre: server umount lustre-MDT0000 complete [ 1262.033667] LustreError: 39992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.401575] LustreError: 39976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386940 with bad export cookie 3828788583678236066 [ 1264.415209] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1264.419258] LustreError: 39976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1264.616072] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.622396] LustreError: 39991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.643965] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1267.144864] LustreError: 39991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.730783] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1269.737978] Lustre: Skipped 1 previous similar message [ 1270.943556] Lustre: server umount lustre-MDT0001 complete [ 1287.185382] Lustre: server umount lustre-OST0000 complete [ 1306.873980] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1310.434230] Key type lgssc unregistered [ 1310.764889] LNet: 43861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.776086] LNetError: 43861:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.810873] LNet: Removed LNI 192.168.206.107@tcp [ 1311.909474] Key type .llcrypt unregistered [ 1311.921756] Key type ._llcrypt unregistered [ 1338.543790] Key type ._llcrypt registered [ 1338.548699] Key type .llcrypt registered [ 1338.700592] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1340.092113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1340.231236] alg: No test for adler32 (adler32-zlib) [ 1341.256207] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 1341.484762] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1343.176804] Key type lgssc registered [ 1344.406837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1353.029453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1353.059451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.596201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.505516] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1364.960982] LustreError: 45282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.980453] LustreError: 45282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1369.680181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1370.231673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1376.040263] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1386.036585] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1393.754449] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1402.098043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1402.422411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.819741] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1418.353715] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1425.008279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1430.499954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1430.513429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1430.519617] Lustre: lustre-OST0000: Not available for connect from 192.168.206.7@tcp (stopping) [ 1430.534485] Lustre: Skipped 1 previous similar message [ 1431.526132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1431.537878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1431.560414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1433.588767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1433.600625] Lustre: Skipped 1 previous similar message [ 1435.599374] Lustre: lustre-OST0000: Not available for connect from 192.168.206.7@tcp (stopping) [ 1436.541452] Lustre: server umount lustre-OST0000 complete [ 1457.121562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1457.135104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1457.148134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1460.812819] Lustre: server umount lustre-MDT0000 complete [ 1463.266408] LustreError: 45282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1463.283629] LustreError: 45282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1464.160084] LustreError: 45262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387140 with bad export cookie 8510616763232802053 [ 1464.168569] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1464.170892] LustreError: 45262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1470.596770] Lustre: server umount lustre-MDT0001 complete [ 1482.130604] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1484.956171] Key type lgssc unregistered [ 1485.408900] LNet: 48990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.429209] LNetError: 48990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1485.458704] LNet: Removed LNI 192.168.206.107@tcp [ 1486.552059] Key type .llcrypt unregistered [ 1486.560236] Key type ._llcrypt unregistered [ 1516.253926] Key type ._llcrypt registered [ 1516.256841] Key type .llcrypt registered [ 1516.416796] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1518.508821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1518.556401] alg: No test for adler32 (adler32-zlib) [ 1519.825301] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 1520.101673] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1521.816261] Key type lgssc registered [ 1522.887981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1530.295541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1530.308413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.862185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.382233] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1541.096609] LustreError: 50408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.129476] LustreError: 50408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1546.211487] LustreError: 50409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.248373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1548.850418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1554.891883] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1563.860510] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1569.924037] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1578.570503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1579.068072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.565944] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1595.549475] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1602.614384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1611.824581] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1615.355701] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1627.362899] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-22 4h28m21s [ 1628.928733] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1632.228573] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.7@tcp (stopping) [ 1633.249382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1633.257913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1633.265199] Lustre: Skipped 2 previous similar messages [ 1633.283513] Lustre: Skipped 2 previous similar messages [ 1634.341086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1634.352564] Lustre: Skipped 3 previous similar messages [ 1637.110867] Lustre: server umount lustre-MDT0000 complete [ 1638.347107] LustreError: 50404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.254885] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1640.420332] LustreError: 50409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.445763] LustreError: 50409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1644.485639] LustreError: 50403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1646.974351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1647.110878] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1647.416192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1647.483769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1648.424522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1651.608628] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1655.038059] Lustre: DEBUG MARKER: Wait for df (36497) ... [ 1656.655844] Lustre: DEBUG MARKER: done [ 1662.952293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1662.957213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.967975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1663.979753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.095159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1669.100445] Lustre: Skipped 3 previous similar messages [ 1675.232166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1675.409412] Lustre: server umount lustre-OST0000 complete [ 1679.329260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1679.331672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1679.341578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1679.341586] Lustre: Skipped 2 previous similar messages [ 1679.361613] Lustre: Skipped 1 previous similar message [ 1685.478473] LustreError: 50408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1685.510548] LustreError: 50408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1685.700683] Lustre: server umount lustre-MDT0000 complete [ 1690.123298] LustreError: 50389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387366 with bad export cookie 4671234200440061939 [ 1690.126329] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1690.131702] LustreError: 50389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1690.512880] Lustre: server umount lustre-MDT0001 complete [ 1702.035727] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1704.558739] Key type lgssc unregistered [ 1704.884797] LNet: 55355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.897271] LNetError: 55355:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.912683] LNet: Removed LNI 192.168.206.107@tcp [ 1705.685739] Key type .llcrypt unregistered [ 1705.689554] Key type ._llcrypt unregistered [ 1742.779636] Key type ._llcrypt registered [ 1742.782774] Key type .llcrypt registered [ 1742.886229] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1743.913993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1743.984507] alg: No test for adler32 (adler32-zlib) [ 1744.925766] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 1745.119688] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1746.762114] Key type lgssc registered [ 1747.977611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1756.165061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1756.177672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.466277] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1757.493063] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1757.711758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.134930] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1770.901223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.018945] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1771.062658] Lustre: 57565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1771.430334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1775.926880] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1784.169587] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1789.915700] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1796.558757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1796.785838] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1796.958671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.791240] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1806.014389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1809.425982] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1816.625809] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-22 4h31m31s [ 1818.192820] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1825.028909] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1827.294560] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1827.348881] LustreError: 57570:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1847.812734] Lustre: lustre-MDT0000: Client ae759ef4-2892-4ab3-ba29-968280a049f6 (at 192.168.206.7@tcp) reconnecting [ 1867.432778] LustreError: 57570:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1869.481253] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1872.143591] Lustre: DEBUG MARKER: done [ 1882.594205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1882.606949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.623930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.107671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.132185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.137717] Lustre: Skipped 1 previous similar message [ 1888.239438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1888.255939] Lustre: Skipped 1 previous similar message [ 1893.344219] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1893.355772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1893.366229] Lustre: Skipped 1 previous similar message [ 1893.523555] Lustre: server umount lustre-OST0000 complete [ 1898.470225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1898.474072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1898.496611] Lustre: Skipped 1 previous similar message [ 1898.545085] Lustre: Skipped 1 previous similar message [ 1910.755933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1910.767054] Lustre: Skipped 5 previous similar messages [ 1912.288300] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1912.559313] Lustre: server umount lustre-MDT0000 complete [ 1917.417446] LustreError: 57556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387593 with bad export cookie 837540691379811511 [ 1917.428332] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1917.435753] LustreError: 57556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1918.161345] Lustre: server umount lustre-MDT0001 complete [ 1931.013338] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1933.811396] Key type lgssc unregistered [ 1934.204660] LNet: 61895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1934.217233] LNetError: 61895:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1934.239496] LNet: Removed LNI 192.168.206.107@tcp [ 1935.179974] Key type .llcrypt unregistered [ 1935.183059] Key type ._llcrypt unregistered [ 1977.417494] Key type ._llcrypt registered [ 1977.420445] Key type .llcrypt registered [ 1977.524786] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1978.878027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1979.023312] alg: No test for adler32 (adler32-zlib) [ 1980.210917] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 1980.502346] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1982.194648] Key type lgssc registered [ 1983.327086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1986.864261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1986.888172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.193249] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1988.218583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1988.413891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.283651] Lustre: Failing over lustre-MDT0000 [ 1989.554391] Lustre: server umount lustre-MDT0000 complete [ 1995.556599] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 04:34:30 (1787387670) [ 2002.655824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2003.250742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.702244] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2016.996955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.221317] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2017.272528] Lustre: 64695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2017.537677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2023.043874] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2031.242144] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2037.060770] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2044.422176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2044.668648] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2044.812491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.454308] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2053.282255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2059.130796] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2065.318946] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2072.820908] Lustre: server umount lustre-MDT0000 complete [ 2074.089465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2074.094468] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.139641] Lustre: Skipped 1 previous similar message [ 2077.611956] LustreError: 64688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387753 with bad export cookie 12118352967258157094 [ 2077.617741] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2077.624428] LustreError: 64688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2077.933626] Lustre: server umount lustre-MDT0001 complete [ 2082.452376] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2088.662121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2095.520935] Lustre: 63619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387755/real 1787387755] req@000000002e802acd x1874211798817536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787387771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2095.543985] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.572314] Lustre: Skipped 1 previous similar message [ 2098.657212] Lustre: 63619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387758/real 1787387758] req@00000000b2ecc6c5 x1874211798817664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787387774 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2099.008915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.704193] Lustre: 63617:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387760/real 1787387760] req@000000007eb90213 x1874211798817792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787387776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2103.506213] LustreError: 66544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.516308] LustreError: 66544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2103.604613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2103.738979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2104.673842] Lustre: 63616:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387763/real 1787387763] req@00000000976b9b14 x1874211798818048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787387779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2105.644482] LustreError: 68464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.664611] LustreError: 68464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2108.708213] Lustre: 63618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387768/real 1787387768] req@000000009974dfb2 x1874211798818176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787387784 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2109.722465] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2110.963533] LustreError: 68462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2111.009758] LustreError: 68462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2114.020461] LustreError: 68462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.024423] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2114.037066] LustreError: 68462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2119.141805] LustreError: 68464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.164948] LustreError: 68464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2120.135095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.754091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2125.161176] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2125.814452] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2132.859213] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2138.814691] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2151.396750] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.406575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2151.423081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2151.438205] Lustre: Skipped 2 previous similar messages [ 2152.889665] Lustre: server umount lustre-MDT0000 complete [ 2156.517427] LustreError: 68464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2156.545648] LustreError: 68464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2156.884362] LustreError: 64688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387832 with bad export cookie 12118352967258159369 [ 2156.886030] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2156.901625] LustreError: 64688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2157.382309] Lustre: server umount lustre-MDT0001 complete [ 2163.822756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2172.885406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2176.993080] Lustre: 63619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387837/real 1787387837] req@0000000030d20f1d x1874211798846464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787387853 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2177.011453] Lustre: 63619:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2177.021760] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.031705] Lustre: Skipped 2 previous similar messages [ 2181.154494] LustreError: 63615:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000014b47869 x1874211798847232/t0(0) o250->MGC192.168.206.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2181.523213] LustreError: 66544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2181.604574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2181.675035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2185.858840] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2193.729607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.888642] Lustre: 63618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387853/real 1787387853] req@0000000085260b51 x1874211798846976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787387869 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2193.906737] Lustre: 63618:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2198.087203] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2199.537714] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2204.749809] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2206.831974] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2210.855691] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2219.496658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2219.520243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2219.554206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.564521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.684633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.697489] Lustre: Skipped 3 previous similar messages [ 2232.288265] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2232.544153] Lustre: server umount lustre-MDT0000 complete [ 2232.814895] LustreError: 66544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.832710] LustreError: 66544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2236.499148] LustreError: 64688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387912 with bad export cookie 12118352967258160748 [ 2236.500729] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2236.506912] LustreError: 64688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2236.920434] Lustre: server umount lustre-MDT0001 complete [ 2242.980591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2246.123755] Lustre: DEBUG MARKER: files compared the same [ 2258.553351] Lustre: server umount lustre-OST0000 complete [ 2274.761736] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 2277.545895] Key type lgssc unregistered [ 2277.808923] LNet: 74424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2277.824595] LNetError: 74424:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2277.837252] LNet: Removed LNI 192.168.206.107@tcp [ 2278.697829] Key type .llcrypt unregistered [ 2278.699954] Key type ._llcrypt unregistered [ 2300.391665] Key type ._llcrypt registered [ 2300.398922] Key type .llcrypt registered [ 2300.542896] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2301.380773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2301.400684] alg: No test for adler32 (adler32-zlib) [ 2302.410320] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 2302.667615] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 2304.304342] Key type lgssc registered [ 2305.262735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2311.910696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2311.928321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.283573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.497212] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug +malloc all [ 2323.432026] LustreError: 75841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.454188] LustreError: 75841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2326.396829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.860923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2331.514389] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug +malloc all [ 2340.269891] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2346.556555] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2353.375768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2353.770970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.903387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2360.683208] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug +malloc all [ 2367.836843] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2381.802734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2381.830944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2381.833441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2381.854513] Lustre: Skipped 1 previous similar message [ 2386.928786] LustreError: 77663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.951254] LustreError: 77663:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2387.722962] Lustre: server umount lustre-OST0000 complete [ 2393.569423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2393.578724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.590239] Lustre: Skipped 1 previous similar message [ 2393.606962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2397.155798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2397.166984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2399.098992] Lustre: server umount lustre-MDT0000 complete [ 2402.276894] LustreError: 76849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.297362] LustreError: 76849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2404.303698] LustreError: 75823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388080 with bad export cookie 15528377167923064134 [ 2404.312405] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2404.315389] LustreError: 75823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2404.679534] Lustre: server umount lustre-MDT0001 complete [ 2417.110664] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 2419.780738] Key type lgssc unregistered [ 2420.140510] LNet: 79391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2420.146336] LNetError: 79391:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.171641] LNet: Removed LNI 192.168.206.107@tcp [ 2420.727978] Key type .llcrypt unregistered [ 2420.729769] Key type ._llcrypt unregistered [ 2446.357818] Key type ._llcrypt registered [ 2446.361575] Key type .llcrypt registered [ 2446.524553] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2447.831181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2447.983865] alg: No test for adler32 (adler32-zlib) [ 2449.107498] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 2449.396165] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 2451.040372] Key type lgssc registered [ 2452.229540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2460.655204] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2460.672530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2476.192423] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2481.312343] LustreError: 80784:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.107@tcp: failed processing log, type 4: rc = -110 [ 2508.000380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2514.777676] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2523.436589] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2532.927681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.174640] Lustre: *** cfs_fail_loc=706, val=0*** [ 2533.592247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.715501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2538.991882] LustreError: 80809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2539.038942] LustreError: 80809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2540.407399] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2542.072295] LustreError: 80810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.103092] LustreError: 80810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2547.181234] LustreError: 81870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.217084] LustreError: 81870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2549.931900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.241881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2554.660308] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2562.041532] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2568.771392] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2570.729648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2570.736382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2570.749886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2575.846417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2575.862364] Lustre: Skipped 1 previous similar message [ 2576.959534] Lustre: server umount lustre-OST0000 complete [ 2586.083341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2586.093176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2586.095172] Lustre: Skipped 2 previous similar messages [ 2586.097011] Lustre: Skipped 1 previous similar message [ 2587.480589] Lustre: server umount lustre-MDT0000 complete [ 2591.193862] LustreError: 80790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388267 with bad export cookie 10014700105544723195 [ 2591.200602] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2591.203156] LustreError: 80790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2591.593874] Lustre: server umount lustre-MDT0001 complete [ 2603.713719] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 2606.797461] Key type lgssc unregistered [ 2607.086642] LNet: 84401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2607.095814] LNetError: 84401:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2607.117151] LNet: Removed LNI 192.168.206.107@tcp [ 2607.916852] Key type .llcrypt unregistered [ 2607.919391] Key type ._llcrypt unregistered [ 2632.391844] Key type ._llcrypt registered [ 2632.393895] Key type .llcrypt registered [ 2632.479683] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2633.235814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2633.264603] alg: No test for adler32 (adler32-zlib) [ 2634.305537] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 2634.500580] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 2636.144852] Key type lgssc registered [ 2637.298825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2644.864325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2644.884770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.825825] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2658.922886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.457802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2663.602789] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2664.954302] LustreError: 86508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2664.979889] LustreError: 86508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2669.101945] LustreError: 86511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.122167] LustreError: 86511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2671.798137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2672.257137] LustreError: 86508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.287891] LustreError: 86508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2672.307824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.731579] LustreError: 86512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.753717] LustreError: 86512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2679.614194] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2682.850856] LustreError: 86511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.862921] LustreError: 86511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2688.626320] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2688.640734] Lustre: MGS: Client 2e13a26c-5b61-46df-a285-d19d78d78f08 (at 0@lo) reconnecting [ 2688.691336] LustreError: 86522:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2688.988140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.098570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2693.791792] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2709.475345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.481986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2709.488651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2709.489620] Lustre: Skipped 1 previous similar message [ 2713.512856] Lustre: server umount lustre-OST0000 complete [ 2719.718636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.718756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2719.742607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2719.742615] Lustre: Skipped 1 previous similar message [ 2724.217967] Lustre: server umount lustre-MDT0000 complete [ 2724.834246] LustreError: 88054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.856062] LustreError: 88054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2728.721630] Lustre: server umount lustre-MDT0001 complete [ 2733.005602] Lustre: server umount MGS complete [ 2744.360287] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:46:59 (1787388419) [ 2770.572641] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 2781.065412] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2815.768581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.718317] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2830.452593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2830.678406] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2830.715828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2830.773530] Lustre: lustre-MDT0001: new disk, initializing [ 2830.830892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2830.853320] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2830.867033] Lustre: 93676:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2830.885628] LustreError: 93676:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2835.838282] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2847.058104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2847.320584] Lustre: lustre-OST0000: new disk, initializing [ 2847.324944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2847.333356] Lustre: 94441:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2847.393518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2853.505765] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2854.960303] Lustre: 94813:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000085260b51 x1874212484587392/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787388547 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2863.386966] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2870.620256] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2870.635776] Lustre: MGS: Client 86401379-e2f4-45ee-95da-6348d0ba3598 (at 0@lo) reconnecting [ 2870.718574] Lustre: 95486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2870.740546] LustreError: 93662:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2870.766386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2870.820915] Lustre: lustre-MDT0000: new disk, initializing [ 2870.919272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2870.942461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2870.952950] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2871.979288] Lustre: 95510:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000004dc1ee90 x1874212484594304/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787388564 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2876.207405] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2877.042050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2877.055803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2877.191969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2902.503546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2902.506300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2902.514701] Lustre: Skipped 2 previous similar messages [ 2902.523649] Lustre: Skipped 2 previous similar messages [ 2907.620968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2907.629400] Lustre: Skipped 1 previous similar message [ 2912.743872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2912.754900] Lustre: Skipped 1 previous similar message [ 2915.296151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2915.420448] Lustre: server umount lustre-OST0000 complete [ 2922.984291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2922.993143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2922.996425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2923.029550] Lustre: Skipped 1 previous similar message [ 2924.961813] Lustre: server umount lustre-MDT0000 complete [ 2928.107558] LustreError: 93647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2928.134305] LustreError: 93647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2934.884619] Lustre: server umount lustre-MDT0001 complete [ 2938.631919] Lustre: server umount MGS complete [ 2949.674682] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:50:24 (1787388624) [ 2965.460158] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 2967.882050] Key type lgssc unregistered [ 2968.134529] LNet: 98006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2968.139656] LNetError: 98006:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2968.163214] LNet: Removed LNI 192.168.206.107@tcp [ 2969.069873] Key type .llcrypt unregistered [ 2969.072149] Key type ._llcrypt unregistered [ 2983.395534] Key type ._llcrypt registered [ 2983.400198] Key type .llcrypt registered [ 2983.511879] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2984.280964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2984.293096] alg: No test for adler32 (adler32-zlib) [ 2985.344787] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 2985.596461] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 2987.248904] Key type lgssc registered [ 2988.669489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2996.418708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2996.449393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2997.923495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3001.936505] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3007.471257] LustreError: 99210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.500493] LustreError: 99210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3011.474369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3011.716994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3017.717865] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3025.302558] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3030.650337] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3037.628875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3038.073396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3042.185860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3043.869530] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3050.690196] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3061.732689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 3061.734437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.745410] LustreError: Skipped 1 previous similar message [ 3061.753303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3061.795558] Lustre: Skipped 1 previous similar message [ 3066.850428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3066.855214] Lustre: Skipped 1 previous similar message [ 3067.721341] Lustre: server umount lustre-OST0000 complete [ 3071.971507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3071.988290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3071.997016] Lustre: Skipped 2 previous similar messages [ 3077.092945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3077.116933] Lustre: Skipped 1 previous similar message [ 3082.209505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3082.224727] Lustre: Skipped 1 previous similar message [ 3085.280397] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3085.499613] Lustre: server umount lustre-MDT0000 complete [ 3087.330735] LustreError: 100151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3089.654772] LustreError: 99191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388765 with bad export cookie 6058568554909563274 [ 3089.661070] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3089.674324] LustreError: 99191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3089.703830] LustreError: 99937:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3089.720742] LustreError: 99937:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3090.170947] Lustre: server umount lustre-MDT0001 complete [ 3100.866932] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3104.126555] Key type lgssc unregistered [ 3104.408668] LNet: 102744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3104.415357] LNetError: 102744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3104.438063] LNet: Removed LNI 192.168.206.107@tcp [ 3105.435431] Key type .llcrypt unregistered [ 3105.440373] Key type ._llcrypt unregistered [ 3122.128619] Key type ._llcrypt registered [ 3122.134153] Key type .llcrypt registered [ 3122.234151] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3123.413557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3123.493086] alg: No test for adler32 (adler32-zlib) [ 3124.495945] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 3124.798365] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3126.536199] Key type lgssc registered [ 3127.713202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3133.164266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3133.172076] LustreError: 103843:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3133.182599] LustreError: 103844:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3133.188712] LustreError: 103843:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 3133.194858] LustreError: 103843:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3133.202411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3133.210052] LustreError: 103843:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3134.665236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3141.331928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.563983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3145.548901] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3150.843961] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing lsmod [ 3153.595807] LustreError: 105048:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3153.614339] LustreError: 105049:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3153.619715] LustreError: 105048:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3153.627907] LustreError: 105048:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3153.639715] LustreError: 105048:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3153.655381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3154.055488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3156.961556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3156.972221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.996363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3159.524203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3159.537698] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.553852] Lustre: Skipped 1 previous similar message [ 3164.652198] LustreError: 103881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3164.672043] LustreError: 103881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3164.746709] Lustre: server umount lustre-MDT0000 complete [ 3166.699522] LustreError: 103877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3166.716251] LustreError: 103877:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3168.339465] LustreError: 103858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388844 with bad export cookie 12083080880983294764 [ 3168.343083] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3168.347718] LustreError: 103858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3168.760272] LustreError: 104312:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000744ea7d3 x1874212998369280/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3168.894486] Lustre: server umount lustre-MDT0001 complete [ 3182.758270] Lustre: server umount lustre-OST0000 complete [ 3190.232909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3190.678813] LustreError: 106115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.747452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.227057] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3195.876972] LustreError: 106116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3200.036765] LustreError: 106115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3205.364310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3205.624100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3210.272983] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3217.283301] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3223.509545] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3229.925276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3230.182929] LustreError: 107937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.195494] LustreError: 107937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3230.279128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3236.350663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3236.586727] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3243.278801] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3250.303725] Lustre: server umount lustre-OST0000 complete [ 3256.802624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3256.803325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3256.813502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3256.822771] Lustre: Skipped 1 previous similar message [ 3256.835067] Lustre: Skipped 3 previous similar messages [ 3261.925485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3261.934800] Lustre: Skipped 1 previous similar message [ 3263.007246] Lustre: server umount lustre-MDT0000 complete [ 3266.019358] Lustre: 103245:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388926/real 1787388926] req@00000000f3802f89 x1874212998394752/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787388942 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3266.039112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.056297] LustreError: 106115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3266.072337] LustreError: 106115:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3266.466247] LustreError: 106095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388942 with bad export cookie 12083080880983295471 [ 3266.474989] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3266.477213] LustreError: 106095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3266.804692] Lustre: server umount lustre-MDT0001 complete [ 3276.483058] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3278.848785] Key type lgssc unregistered [ 3279.103705] LNet: 109648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3279.108520] LNetError: 109648:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3279.153913] LNet: Removed LNI 192.168.206.107@tcp [ 3279.893084] Key type .llcrypt unregistered [ 3279.894934] Key type ._llcrypt unregistered [ 3298.953622] Key type ._llcrypt registered [ 3298.956119] Key type .llcrypt registered [ 3299.029583] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3299.944583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3300.026953] alg: No test for adler32 (adler32-zlib) [ 3301.032745] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 3301.247500] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3302.944153] Key type lgssc registered [ 3303.780179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3309.583232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3309.601850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.011311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.709349] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3321.315820] LustreError: 111045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.335420] LustreError: 111045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3321.970462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3322.375671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3326.225738] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3334.031040] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3339.268683] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3345.739683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3346.043671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.969867] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3356.277229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3359.180681] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3365.197172] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3368.421555] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3373.025461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3373.034760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3373.074725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.124619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3377.127620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.136276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3377.160832] Lustre: Skipped 1 previous similar message [ 3378.870806] Lustre: server umount lustre-OST0000 complete [ 3383.780764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3383.794321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3383.810643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.363330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.367791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.258944] Lustre: server umount lustre-MDT0000 complete [ 3392.401038] LustreError: 111027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389068 with bad export cookie 4184211362091220842 [ 3392.402245] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3392.413643] LustreError: 111027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3392.805302] Lustre: server umount lustre-MDT0001 complete [ 3402.935539] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3405.382423] Key type lgssc unregistered [ 3405.594189] LNet: 114686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3405.598911] LNetError: 114686:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3406.640838] LNet: Removed LNI 192.168.206.107@tcp [ 3407.605362] Key type .llcrypt unregistered [ 3407.607973] Key type ._llcrypt unregistered [ 3422.948295] Key type ._llcrypt registered [ 3422.950813] Key type .llcrypt registered [ 3423.063580] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3423.879566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3424.044543] alg: No test for adler32 (adler32-zlib) [ 3425.059765] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 3425.345840] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3427.050864] Key type lgssc registered [ 3428.135683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3434.906880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3434.925967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.326498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.702565] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3446.772294] LustreError: 115904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.800714] LustreError: 115904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3449.436746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.809299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3454.117148] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3460.846590] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3465.802160] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3471.672141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3472.052716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3477.847362] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3481.278957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3484.652295] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3491.812967] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3491.816193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3491.822411] Lustre: Skipped 1 previous similar message [ 3491.832853] Lustre: Skipped 1 previous similar message [ 3496.933432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3497.857726] Lustre: server umount lustre-OST0000 complete [ 3501.032756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3501.039210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.054205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3501.058582] Lustre: Skipped 1 previous similar message [ 3502.061860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.169271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3507.182772] Lustre: Skipped 3 previous similar messages [ 3512.294603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3512.298396] Lustre: Skipped 1 previous similar message [ 3515.362237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3515.475788] Lustre: server umount lustre-MDT0000 complete [ 3517.409855] LustreError: 115900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3517.427467] LustreError: 115900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3519.117336] LustreError: 115885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389195 with bad export cookie 11284526914675541696 [ 3519.122671] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3519.132960] LustreError: 115885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3519.418462] Lustre: server umount lustre-MDT0001 complete [ 3529.197407] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3532.143187] Key type lgssc unregistered [ 3532.440163] LNet: 119437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3532.452719] LNetError: 119437:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3532.472969] LNet: Removed LNI 192.168.206.107@tcp [ 3533.430426] Key type .llcrypt unregistered [ 3533.432651] Key type ._llcrypt unregistered [ 3560.760958] Key type ._llcrypt registered [ 3560.768626] Key type .llcrypt registered [ 3560.909374] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3562.090620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3562.179304] alg: No test for adler32 (adler32-zlib) [ 3563.216563] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 3563.443228] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3565.128179] Key type lgssc registered [ 3566.207632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3573.714609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3573.745510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3575.300655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3578.928950] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3585.506822] LustreError: 120854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3585.528897] LustreError: 120854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3586.334951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.541252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3589.911514] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3596.767726] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3602.360970] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3607.869138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3608.213148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3614.036401] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3619.947681] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3620.541261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3629.921743] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3632.919996] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3632.928791] Lustre: Skipped 2 previous similar messages [ 3636.887496] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3636.892252] Lustre: Skipped 2 previous similar messages [ 3640.368591] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3640.387957] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3647.734970] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.7@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3647.744257] Lustre: Skipped 2 previous similar messages [ 3650.166908] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.7@tcp 192.168.0.[2,10]@tcp [ 3653.486244] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.7/24@tcp in log lustre-MDT0000 [ 3653.491947] Lustre: Skipped 2 previous similar messages [ 3657.392094] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.7/24@tcp [ 3657.397924] Lustre: Skipped 1 previous similar message [ 3662.817992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3662.827363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3662.839630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3664.873708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3664.880323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3664.881618] LustreError: Skipped 1 previous similar message [ 3664.900722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3666.913701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3668.764352] Lustre: server umount lustre-OST0000 complete [ 3672.034356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.037705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3672.041572] Lustre: Skipped 1 previous similar message [ 3672.052403] Lustre: Skipped 2 previous similar messages [ 3677.154317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3677.160314] Lustre: Skipped 1 previous similar message [ 3686.368385] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3686.522872] Lustre: server umount lustre-MDT0000 complete [ 3687.401421] LustreError: 121962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.423238] LustreError: 121962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3689.344720] LustreError: 120834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389365 with bad export cookie 9229126183265285017 [ 3689.346823] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3689.355385] LustreError: 120834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3689.676754] Lustre: server umount lustre-MDT0001 complete [ 3699.575210] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3702.065706] Key type lgssc unregistered [ 3702.262144] LNet: 125654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3702.274312] LNetError: 125654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3702.286850] LNet: Removed LNI 192.168.206.107@tcp [ 3702.958453] Key type .llcrypt unregistered [ 3702.961409] Key type ._llcrypt unregistered [ 3730.827891] Key type ._llcrypt registered [ 3730.846240] Key type .llcrypt registered [ 3731.000893] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3732.018633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3732.041433] alg: No test for adler32 (adler32-zlib) [ 3733.224136] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 3733.521496] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3735.202542] Key type lgssc registered [ 3736.436606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3742.998200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3743.014422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3744.605494] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3744.638177] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.7/24@tcp [ 3744.647041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3749.712237] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3753.962171] LustreError: 127217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3753.977043] LustreError: 127217:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3757.309284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3757.498099] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3757.521472] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.7/24@tcp [ 3757.586124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3761.484658] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3767.929487] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3772.719887] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3778.731894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3779.013776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3784.275301] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3789.244499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3792.109500] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3802.595177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3802.609580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.632135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3804.643517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3804.644875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3804.669812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3808.808049] Lustre: server umount lustre-OST0000 complete [ 3813.857280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3813.861365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3813.867738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3813.870317] Lustre: Skipped 1 previous similar message [ 3820.008593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3820.015039] Lustre: Skipped 3 previous similar messages [ 3827.168403] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3827.377351] Lustre: server umount lustre-MDT0000 complete [ 3828.199679] LustreError: 129852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3828.213164] LustreError: 129852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3829.221219] LustreError: 127211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3829.238042] LustreError: 127211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3831.026752] LustreError: 127197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389507 with bad export cookie 9348035164959328264 [ 3831.032853] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3831.042815] LustreError: 127197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3831.318311] Lustre: server umount lustre-MDT0001 complete [ 3841.270465] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3843.595224] Key type lgssc unregistered [ 3843.829573] LNet: 130805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3843.842780] LNetError: 130805:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3843.863422] LNet: Removed LNI 192.168.206.107@tcp [ 3844.579276] Key type .llcrypt unregistered [ 3844.581792] Key type ._llcrypt unregistered [ 3869.704852] Key type ._llcrypt registered [ 3869.707314] Key type .llcrypt registered [ 3869.792694] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 3883.653746] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3884.628571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3884.644089] alg: No test for adler32 (adler32-zlib) [ 3885.665586] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 3885.983532] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3887.648166] Key type lgssc registered [ 3888.917357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3921.745765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3921.781921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3923.111822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3923.141366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3923.218401] Lustre: lustre-MDT0000: new disk, initializing [ 3923.318503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3923.348833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3928.140742] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3938.734719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3938.841378] Lustre: 134204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3938.883335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3938.886317] Lustre: Skipped 1 previous similar message [ 3938.960458] Lustre: lustre-MDT0001: new disk, initializing [ 3939.088619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3939.131408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3939.142630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3943.564674] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3952.915574] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3958.133894] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3965.609367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3965.897745] Lustre: lustre-OST0000: new disk, initializing [ 3965.906740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3965.926107] Lustre: 136174:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3966.050225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3972.314818] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3973.203415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3973.214327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3973.285608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3980.817758] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3984.872401] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3985.099592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3988.683779] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3988.873390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3994.085208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3994.088077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.095877] LustreError: Skipped 1 previous similar message [ 3994.106269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3994.116665] Lustre: Skipped 1 previous similar message [ 3996.546442] Lustre: server umount lustre-OST0000 complete [ 4000.739647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4000.746365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.757334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4000.763390] Lustre: Skipped 1 previous similar message [ 4004.322474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4004.334702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4004.352120] Lustre: Skipped 1 previous similar message [ 4005.948025] Lustre: server umount lustre-MDT0000 complete [ 4009.405250] LustreError: 135000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389685 with bad export cookie 2532548149221114175 [ 4009.407117] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4009.416098] LustreError: 135000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4009.766786] Lustre: server umount lustre-MDT0001 complete [ 4017.154778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4017.603371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4021.880496] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4022.760135] LustreError: 138388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4022.780898] LustreError: 138388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4027.879057] LustreError: 138389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4030.360792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.658487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4034.652238] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4041.758783] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4047.138931] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4053.421086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4053.706387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4059.281672] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4065.890911] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4070.181199] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4070.442244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4074.125699] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4074.275634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4081.991834] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4082.147035] Lustre: lustre-OST0001: new disk, initializing [ 4082.150584] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4082.157071] Lustre: 141761:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4082.227297] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4087.734570] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4090.455165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4090.472429] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4090.578236] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4095.137293] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4095.291827] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4098.788322] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4098.953961] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4103.552833] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff954d07ab3800.ost_server_uuid 50 [ 4104.719582] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff954d07ab3800.ost_server_uuid in FULL state after 0 sec [ 4114.095641] Lustre: server umount lustre-OST0001 complete [ 4116.455866] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4116.467265] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4116.470702] LustreError: 140978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4116.490106] LustreError: 140978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4117.477685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4119.523709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4119.538719] LustreError: Skipped 1 previous similar message [ 4121.569848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4121.576161] Lustre: Skipped 1 previous similar message [ 4121.579267] LustreError: 140210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4121.597095] LustreError: 140210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4126.695713] LustreError: 140209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4126.715275] LustreError: 140209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4131.296418] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4131.518690] Lustre: server umount lustre-OST0000 complete [ 4136.931918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4136.943131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4136.949951] Lustre: Skipped 4 previous similar messages [ 4136.959430] Lustre: Skipped 2 previous similar messages [ 4149.216687] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4149.341676] Lustre: server umount lustre-MDT0000 complete [ 4152.290278] LustreError: 138383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4152.306429] LustreError: 138383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4152.753574] LustreError: 140232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389828 with bad export cookie 2532548149221115050 [ 4152.758793] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4152.765353] LustreError: 140232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4153.116468] Lustre: server umount lustre-MDT0001 complete [ 4172.626410] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 4175.285676] Key type lgssc unregistered [ 4175.601338] LNet: 144764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4175.618390] LNetError: 144764:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4175.636996] LNet: Removed LNI 192.168.206.107@tcp [ 4176.393271] Key type .llcrypt unregistered [ 4176.397284] Key type ._llcrypt unregistered [ 4211.610824] Key type ._llcrypt registered [ 4211.614768] Key type .llcrypt registered [ 4211.696757] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 4224.843177] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 4225.882323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4225.994044] alg: No test for adler32 (adler32-zlib) [ 4226.958173] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 4227.214494] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 4228.856226] Key type lgssc registered [ 4229.942475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4261.312662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4261.344158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4262.619838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4262.653492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4262.780277] Lustre: lustre-MDT0000: new disk, initializing [ 4262.872050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4262.888681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4266.906434] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4276.940419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4277.075373] Lustre: 148913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4277.124252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4277.127739] Lustre: Skipped 1 previous similar message [ 4277.216465] Lustre: lustre-MDT0001: new disk, initializing [ 4277.290429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4277.355216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4277.363742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4281.414540] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4289.854851] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4295.535497] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4302.257715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4302.537890] Lustre: lustre-OST0000: new disk, initializing [ 4302.542331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4302.548394] Lustre: 150879:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4302.633995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4309.133373] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4311.090956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4311.104919] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4311.199923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4316.729703] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4320.099329] Lustre: Failing over lustre-OST0000 [ 4320.230550] Lustre: server umount lustre-OST0000 complete [ 4337.120166] Lustre: 147135:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787389997/real 1787389997] req@000000006d5b7553 x1874214154959616/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787390013 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4337.120767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.144615] Lustre: 147135:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4337.167311] Lustre: Skipped 1 previous similar message [ 4338.535564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4338.660432] Lustre: 147137:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787389998/real 1787389998] req@000000000bab7d34 x1874214154960512/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787390014 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4338.824735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4338.837342] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4342.928878] Lustre: 147137:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390002/real 1787390002] req@000000006fec567f x1874214154961536/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787390018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4343.762690] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4344.236674] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4347.913924] Lustre: Failing over lustre-MDT0000 [ 4348.146525] Lustre: server umount lustre-MDT0000 complete [ 4348.898450] LustreError: 148920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4348.898600] Lustre: 147134:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390008/real 1787390008] req@000000007011e266 x1874214154962688/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787390024 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4348.919036] LustreError: 148920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4348.927882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4348.927905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.952699] Lustre: 147134:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4353.991735] LustreError: 151810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.111967] LustreError: 151810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.128961] LustreError: 151810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4364.236620] LustreError: 152802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4364.257593] Lustre: 147137:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390024/real 1787390024] req@00000000104fd83b x1874214154971520/t0(0) o400->MGC192.168.206.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787390040 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4364.287142] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4366.077018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4369.379036] LustreError: 148924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4369.401686] LustreError: 148924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4374.497117] LustreError: 147133:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b8f0d04c x1874214154975104/t0(0) o250->MGC192.168.206.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4374.711124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4374.761144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4374.805935] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4375.499506] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4379.053982] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4380.146674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4380.215840] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4380.250823] Lustre: Skipped 1 previous similar message [ 4380.321161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4390.378867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4390.379390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4390.382279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4390.394107] Lustre: Skipped 3 previous similar messages [ 4393.351267] Lustre: server umount lustre-OST0000 complete [ 4400.613968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4400.619578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4400.641481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4400.641488] Lustre: Skipped 1 previous similar message [ 4402.912642] Lustre: server umount lustre-MDT0000 complete [ 4405.737201] LustreError: 152802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4405.757889] LustreError: 152802:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4406.599352] LustreError: 148903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390082 with bad export cookie 747220992914716935 [ 4406.606447] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4406.619879] LustreError: 148903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4407.014746] Lustre: server umount lustre-MDT0001 complete [ 4417.948284] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 4421.101770] Key type lgssc unregistered [ 4421.417403] LNet: 154544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4421.426471] LNetError: 154544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4421.447899] LNet: Removed LNI 192.168.206.107@tcp [ 4422.290902] Key type .llcrypt unregistered [ 4422.293067] Key type ._llcrypt unregistered [ 4460.660570] Key type ._llcrypt registered [ 4460.674319] Key type .llcrypt registered [ 4460.797276] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 4461.586246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4461.648064] alg: No test for adler32 (adler32-zlib) [ 4462.653972] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 4462.963113] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 4464.744396] Key type lgssc registered [ 4465.934499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4473.866585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4473.887288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4475.186365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.376394] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4485.607079] LustreError: 156071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4485.629138] LustreError: 156071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4487.824708] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4490.724902] LustreError: 156072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4491.677593] LustreError: 156100:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4491.685170] LustreError: 156100:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4491.693366] LustreError: 156100:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4492.016532] Lustre: server umount lustre-MDT0000 complete [ 4505.222629] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 4515.498170] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 4553.650168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4554.024621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4554.049596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4554.151437] Lustre: lustre-MDT0000: new disk, initializing [ 4554.234411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4554.262700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4559.003841] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4571.183516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.349778] Lustre: 159403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4571.433151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4571.439750] Lustre: Skipped 1 previous similar message [ 4571.538545] Lustre: lustre-MDT0001: new disk, initializing [ 4571.666772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4571.719495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4571.730408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4576.077623] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4587.380736] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4593.882968] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4601.430284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4601.804630] Lustre: lustre-OST0000: new disk, initializing [ 4601.811205] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4601.820269] Lustre: 161372:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4601.953622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4608.306871] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4611.235678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4611.254469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4611.377989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4618.686195] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4623.068759] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4624.948931] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 05:18:19 (1787390299) [ 4888.033083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4888.042987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4888.047930] Lustre: Skipped 2 previous similar messages [ 4888.059679] Lustre: Skipped 2 previous similar messages [ 4893.156799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4893.175041] Lustre: Skipped 2 previous similar messages [ 4893.526607] Lustre: server umount lustre-MDT0000 complete [ 4897.322344] LustreError: 163142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390573 with bad export cookie 7866938122354737860 [ 4897.325846] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4897.335272] LustreError: 163142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4898.273064] LustreError: 163152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4898.273597] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4898.307773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4902.315987] LustreError: 159409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4902.319303] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4903.779665] Lustre: server umount lustre-MDT0001 complete [ 4922.336719] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4922.503073] Lustre: server umount lustre-OST0000 complete [ 4931.431943] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 4940.519675] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 4978.138631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4978.783810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4978.885338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4979.073653] Lustre: lustre-MDT0000: new disk, initializing [ 4979.202463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4979.242948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4984.440137] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4995.974743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4996.211829] Lustre: 166041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4996.278647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4996.282197] Lustre: Skipped 1 previous similar message [ 4996.341300] Lustre: lustre-MDT0001: new disk, initializing [ 4996.410156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4996.480989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4996.497907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5001.841784] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5012.071793] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5018.322878] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5024.477305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5024.750463] Lustre: lustre-OST0000: new disk, initializing [ 5024.753292] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5024.760114] Lustre: 168008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5024.842115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5030.564490] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5035.121675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5035.131051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5035.271039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5039.254188] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5043.590154] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5043.845133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5048.834991] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5049.023573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5050.849845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5050.865488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5050.866371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5050.884352] LustreError: Skipped 1 previous similar message [ 5056.872786] Lustre: server umount lustre-OST0000 complete [ 5061.092718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5061.101057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5061.114888] Lustre: Skipped 2 previous similar messages [ 5061.139422] Lustre: Skipped 4 previous similar messages [ 5074.400172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5074.621777] Lustre: server umount lustre-MDT0000 complete [ 5076.451482] LustreError: 166052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5076.476576] LustreError: 166052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5078.020063] LustreError: 166844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390754 with bad export cookie 7866938122354896690 [ 5078.027477] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5078.034231] LustreError: 166844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5078.350354] Lustre: server umount lustre-MDT0001 complete [ 5087.072538] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:26:01 (1787390761) [ 5102.115977] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 5111.972387] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 5143.813950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5144.047510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5144.072284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5144.153663] Lustre: lustre-MDT0000: new disk, initializing [ 5144.216860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5144.241785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5147.975911] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5158.276730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5158.355391] Lustre: 172763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5158.646985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5162.434412] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5170.809248] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5175.708635] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5182.121606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5182.338659] Lustre: lustre-OST0000: new disk, initializing [ 5182.341511] Lustre: Skipped 1 previous similar message [ 5182.344882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5182.349612] Lustre: Skipped 2 previous similar messages [ 5182.353638] Lustre: 174740:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5188.225530] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5192.288601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5192.297154] Lustre: Skipped 1 previous similar message [ 5192.305436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5192.358707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5196.216294] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5202.913549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5202.923840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5202.926601] LustreError: Skipped 1 previous similar message [ 5202.929829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5202.940025] Lustre: Skipped 1 previous similar message [ 5202.968534] Lustre: Skipped 5 previous similar messages [ 5208.800496] Lustre: server umount lustre-OST0000 complete [ 5213.153882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.169603] Lustre: Skipped 1 previous similar message [ 5218.280348] LustreError: 172774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5218.349292] Lustre: server umount lustre-MDT0000 complete [ 5221.640458] LustreError: 172756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390897 with bad export cookie 7866938122354897572 [ 5221.643675] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5221.653235] LustreError: 172756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5221.867871] Lustre: server umount lustre-MDT0001 complete [ 5228.363599] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:28:23 (1787390903) [ 5242.850389] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 5252.136640] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 5287.645406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5287.863684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5287.893959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5288.001880] Lustre: lustre-MDT0000: new disk, initializing [ 5288.079262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5288.086273] Lustre: Skipped 2 previous similar messages [ 5288.111799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5292.538641] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5302.220769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5302.336656] Lustre: 179061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5302.534507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5306.945981] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5315.355334] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5320.952676] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5329.106206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5329.438830] Lustre: 181039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5335.600926] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5337.680666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5337.793271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5344.156689] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5353.443121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5353.446140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5353.460962] LustreError: Skipped 1 previous similar message [ 5353.472126] Lustre: Skipped 1 previous similar message [ 5353.484411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5353.503795] Lustre: Skipped 5 previous similar messages [ 5357.459878] Lustre: server umount lustre-OST0000 complete [ 5368.583792] Lustre: server umount lustre-MDT0000 complete [ 5368.812236] LustreError: 179073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5368.835628] LustreError: 179073:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5373.638735] LustreError: 179052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391049 with bad export cookie 7866938122354898657 [ 5373.655422] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5373.681931] LustreError: 179052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5374.428345] Lustre: server umount lustre-MDT0001 complete [ 5388.120324] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 5391.312323] Key type lgssc unregistered [ 5391.716779] LNet: 182896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5391.725116] LNetError: 182896:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5391.743704] LNet: Removed LNI 192.168.206.107@tcp [ 5392.596812] Key type .llcrypt unregistered [ 5392.600254] Key type ._llcrypt unregistered [ 5416.071804] Key type ._llcrypt registered [ 5416.075971] Key type .llcrypt registered [ 5416.178468] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 5416.891874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5416.950776] alg: No test for adler32 (adler32-zlib) [ 5418.103587] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 5418.279277] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 5419.944525] Key type lgssc registered [ 5421.026522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5429.266730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5429.294397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5430.860756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5435.436736] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5439.976182] LustreError: 184313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5439.992093] LustreError: 184313:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5445.090141] LustreError: 184314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5445.686302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5446.137550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5450.410176] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5457.710790] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5463.427315] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5470.238366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5470.676641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5475.837875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5476.183992] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5482.553903] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5501.412476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5501.412842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5501.427197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5501.436985] Lustre: Skipped 1 previous similar message [ 5501.451852] Lustre: Skipped 1 previous similar message [ 5505.307346] Lustre: server umount lustre-OST0000 complete [ 5511.651561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5511.652944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5511.670165] Lustre: Skipped 1 previous similar message [ 5514.785617] Lustre: server umount lustre-MDT0000 complete [ 5516.773354] LustreError: 184314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5517.899726] LustreError: 184293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391193 with bad export cookie 15823107429078025941 [ 5517.913776] LustreError: 184293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5517.915880] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5518.237670] Lustre: server umount lustre-MDT0001 complete [ 5527.256809] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 5529.555693] Key type lgssc unregistered [ 5529.780641] LNet: 187869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5529.786314] LNetError: 187869:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5529.797625] LNet: Removed LNI 192.168.206.107@tcp [ 5530.608494] Key type .llcrypt unregistered [ 5530.611197] Key type ._llcrypt unregistered [ 5554.364405] Key type ._llcrypt registered [ 5554.367784] Key type .llcrypt registered [ 5554.480630] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 5555.424471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5555.513505] alg: No test for adler32 (adler32-zlib) [ 5556.662428] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 5556.921928] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 5558.616247] Key type lgssc registered [ 5559.650798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5567.582732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5567.609485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.036522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5573.277342] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5579.241405] LustreError: 189289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5579.265463] LustreError: 189289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5581.806250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5582.295696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5587.508753] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5595.849648] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5601.541713] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5609.109456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5609.707079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.841794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5616.072416] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5623.839325] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5635.554244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5635.561345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.575091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5637.097817] Lustre: lustre-OST0000: Not available for connect from 192.168.206.7@tcp (stopping) [ 5637.612356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5637.622580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5640.532410] Lustre: server umount lustre-OST0000 complete [ 5648.980476] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff954d04e62800.ost_server_uuid 50 [ 5650.747698] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5667.297556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5667.302642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5667.322104] Lustre: Skipped 1 previous similar message [ 5667.337309] Lustre: Skipped 3 previous similar messages [ 5671.385833] Lustre: server umount lustre-MDT0000 complete [ 5672.420365] LustreError: 189289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5672.445492] LustreError: 189289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5675.334637] LustreError: 189270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391351 with bad export cookie 16934527537166772021 [ 5675.355150] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5675.360116] LustreError: 189270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5675.829290] Lustre: server umount lustre-MDT0001 complete [ 5685.490802] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 05:36:00 (1787391360) [ 5693.372415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5693.960763] LustreError: 193208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5694.105845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5699.215147] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5699.557091] LustreError: 193209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5699.581905] LustreError: 193209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5704.675285] LustreError: 193208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5707.558709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5708.030585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5713.680802] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5723.585679] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5730.377447] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5737.408330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5737.873225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5739.967981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5744.568878] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5751.946094] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5759.267557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5759.505704] Lustre: lustre-OST0001: new disk, initializing [ 5759.513371] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5759.534669] Lustre: 196049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5759.660860] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5765.707295] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5767.784161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5767.805736] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5767.921971] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5776.611988] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5780.965127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5780.975443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.990450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5785.706236] Lustre: server umount lustre-OST0000 complete [ 5787.599857] LustreError: 195031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5791.916084] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5792.126467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5796.655898] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5796.866815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5804.006310] LustreError: 196057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5804.033921] LustreError: 196057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5812.204067] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5812.209631] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5812.231089] Lustre: Skipped 2 previous similar messages [ 5812.242184] Lustre: Skipped 4 previous similar messages [ 5817.470437] Lustre: server umount lustre-OST0001 complete [ 5822.438807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.460973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5822.470676] Lustre: Skipped 1 previous similar message [ 5828.019392] Lustre: server umount lustre-MDT0000 complete [ 5831.702652] LustreError: 193189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391507 with bad export cookie 16934527537166773092 [ 5831.704800] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5831.723545] LustreError: 193189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5832.394710] Lustre: server umount lustre-MDT0001 complete [ 5858.780145] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 05:38:53 (1787391533) [ 5866.909444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5867.233910] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5867.310243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5867.535630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5872.234175] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5881.991719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5882.191866] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5882.286210] Lustre: 199488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5888.538428] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5897.151764] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5903.658523] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5910.695433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5910.894184] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5916.923498] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5920.931685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5924.339268] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5931.341775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5931.407793] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5931.677424] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5931.682772] Lustre: Skipped 2 previous similar messages [ 5937.340995] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5938.884596] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5945.851129] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5949.416582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5949.431634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5949.436751] Lustre: Skipped 2 previous similar messages [ 5949.449775] Lustre: Skipped 3 previous similar messages [ 5954.848235] Lustre: server umount lustre-OST0000 complete [ 5957.063718] LustreError: 201342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5957.090199] LustreError: 201342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5975.009284] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5977.541340] Lustre: server umount lustre-OST0001 complete [ 5985.259109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.260609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5985.265898] Lustre: Skipped 2 previous similar messages [ 5985.267341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5985.279658] LustreError: Skipped 1 previous similar message [ 5985.282169] Lustre: Skipped 6 previous similar messages [ 5988.366594] Lustre: server umount lustre-MDT0000 complete [ 5993.134403] LustreError: 199478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391669 with bad export cookie 16934527537166774499 [ 5993.137121] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5993.144279] LustreError: 199478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5993.466246] Lustre: server umount lustre-MDT0001 complete [ 6020.779286] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 05:41:35 (1787391695) [ 6039.547213] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 6051.143410] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 6089.097559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6089.428491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6089.451325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6089.528288] Lustre: lustre-MDT0000: new disk, initializing [ 6089.627199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6089.663987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6094.401707] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6104.421954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6104.627945] Lustre: 207588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6104.711522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6104.715938] Lustre: Skipped 1 previous similar message [ 6104.823684] Lustre: lustre-MDT0001: new disk, initializing [ 6104.996459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6105.012334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6110.357424] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6121.920426] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6128.453411] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6136.064755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6136.368683] Lustre: lustre-OST0000: new disk, initializing [ 6136.373957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6136.380659] Lustre: 209557:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6143.266642] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6145.614831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6145.627091] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6145.776874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6153.330992] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6158.193883] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6158.391489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6162.827902] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6163.053583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6166.498472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6166.511121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6166.520563] Lustre: Skipped 1 previous similar message [ 6166.524115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6166.534994] Lustre: Skipped 1 previous similar message [ 6179.296404] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6179.541250] Lustre: server umount lustre-OST0000 complete [ 6197.216185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6197.574927] Lustre: server umount lustre-MDT0000 complete [ 6202.099647] LustreError: 207579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391878 with bad export cookie 16934527537166776284 [ 6202.116030] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6202.129289] LustreError: 207579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6202.756664] Lustre: server umount lustre-MDT0001 complete [ 6214.078098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6214.734086] LustreError: 211777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6214.758901] LustreError: 211777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 6219.255775] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6226.979290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6232.371162] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6239.612487] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6244.994598] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6252.733281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6261.014414] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6267.767432] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6271.847686] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6271.996233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6276.294186] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6276.542081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6280.168064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6280.173880] LustreError: Skipped 1 previous similar message [ 6284.648516] Lustre: server umount lustre-OST0000 complete [ 6291.046110] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6295.224304] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6325.471770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6332.160221] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6333.105155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.129278] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6340.099114] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6342.053499] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6346.184398] Lustre: server umount lustre-OST0000 complete [ 6350.307079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6350.314123] LustreError: Skipped 1 previous similar message [ 6350.316824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6350.329407] Lustre: Skipped 4 previous similar messages [ 6350.332920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6350.339455] Lustre: Skipped 13 previous similar messages [ 6364.129071] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6364.403848] Lustre: server umount lustre-MDT0000 complete [ 6368.643485] LustreError: 215885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392044 with bad export cookie 16934527537166777166 [ 6368.647367] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6368.663450] LustreError: 215885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6369.038743] Lustre: server umount lustre-MDT0001 complete [ 6378.154995] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 05:47:32 (1787392052) [ 6385.246542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6385.684688] LustreError: 217743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6385.704357] LustreError: 217743:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6385.743891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6385.749294] Lustre: Skipped 6 previous similar messages [ 6390.714735] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6398.500345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6403.520217] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6410.610456] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6416.744936] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6423.282745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6429.403928] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6436.379566] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6440.990032] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6441.230913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6445.742661] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6445.955541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6453.414186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6453.586145] Lustre: lustre-OST0001: new disk, initializing [ 6453.589981] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6453.597903] Lustre: 221112:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6459.845567] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6463.147464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6463.162279] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6463.311650] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6470.439315] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6475.513813] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6475.736805] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6480.304798] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6480.501706] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6483.946352] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6488.547782] Lustre: server umount lustre-OST0001 complete [ 6494.721690] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6494.994529] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6500.345953] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6500.515052] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6529.645240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6535.152292] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6535.162783] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6535.174205] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6535.191834] Lustre: Skipped 1 previous similar message [ 6536.335919] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6544.018181] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6545.891393] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6562.272125] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6562.397747] Lustre: server umount lustre-OST0001 complete [ 6586.790456] Lustre: server umount lustre-OST0000 complete [ 6598.085069] Lustre: server umount lustre-MDT0000 complete [ 6602.448269] LustreError: 217723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392278 with bad export cookie 16934527537166778125 [ 6602.451592] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6602.461758] LustreError: 217723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6609.053799] Lustre: server umount lustre-MDT0001 complete [ 6634.061566] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 05:51:48 (1787392308) [ 6640.501573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6640.845129] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6640.876153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6645.594686] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6654.165309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6654.246857] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6654.284784] Lustre: 226360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6654.389465] LustreError: 226371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6654.408318] LustreError: 226371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 36 previous similar messages [ 6659.174727] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6667.082202] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6673.007347] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6679.958990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6680.232359] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6686.608619] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6694.535332] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6704.045982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6704.124230] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6710.471317] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6718.571514] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6720.210401] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6724.581404] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6724.825474] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6729.841308] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6730.052623] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6735.308512] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff954d09fb0000.ost_server_uuid 50 [ 6736.672963] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff954d09fb0000.ost_server_uuid in IDLE state after 0 sec [ 6737.516428] Lustre: Permanently deactivating lustre-OST0001 [ 6737.530382] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6740.542687] Lustre: Permanently reactivating lustre-OST0001 [ 6746.081460] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6746.095748] LustreError: Skipped 2 previous similar messages [ 6746.101411] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6746.132427] Lustre: Skipped 9 previous similar messages [ 6746.140496] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6746.149324] Lustre: Skipped 21 previous similar messages [ 6751.385367] Lustre: server umount lustre-OST0001 complete [ 6770.144208] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6770.456351] Lustre: server umount lustre-OST0000 complete [ 6781.465256] Lustre: server umount lustre-MDT0000 complete [ 6784.930273] LustreError: 230773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392460 with bad export cookie 16934527537166779413 [ 6784.931110] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6784.945136] LustreError: 230773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6785.344790] Lustre: server umount lustre-MDT0001 complete [ 6811.660976] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 05:54:46 (1787392486) [ 6819.330951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6819.519718] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6819.541876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6819.553221] Lustre: Skipped 5 previous similar messages [ 6824.471853] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6833.411375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6833.593869] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6833.648350] Lustre: 233060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6838.427203] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6845.860640] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6851.240465] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6857.881802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6858.187842] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6858.211538] Lustre: Permanently deactivating lustre-OST0000 [ 6858.226947] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6864.333627] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6866.465537] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6866.521054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6866.528285] LustreError: 235402:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 6871.099476] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6878.152698] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6878.198200] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6884.368397] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 6888.716972] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6892.492444] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6894.565193] Lustre: Permanently reactivating lustre-OST0000 [ 6894.588989] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6894.600620] Lustre: Skipped 2 previous similar messages [ 6903.849757] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6903.853415] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6903.871893] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6903.924230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6903.933547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6923.232741] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6923.436073] Lustre: server umount lustre-OST0001 complete [ 6942.177770] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6942.358444] Lustre: server umount lustre-OST0000 complete [ 6952.374114] Lustre: server umount lustre-MDT0000 complete [ 6956.875561] LustreError: 233096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392632 with bad export cookie 16934527537166781065 [ 6956.881814] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6956.918483] LustreError: 233096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6957.382914] Lustre: server umount lustre-MDT0001 complete [ 6967.878194] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 6970.700778] Key type lgssc unregistered [ 6970.974709] LNet: 237940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6970.987655] LNetError: 237940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6972.007947] LNet: Removed LNI 192.168.206.107@tcp [ 6972.849683] Key type .llcrypt unregistered [ 6972.856741] Key type ._llcrypt unregistered [ 6996.201500] Key type ._llcrypt registered [ 6996.205785] Key type .llcrypt registered [ 6996.326849] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 6997.170171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6997.324290] alg: No test for adler32 (adler32-zlib) [ 6998.297360] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 6998.591245] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 7000.248562] Key type lgssc registered [ 7001.445960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7010.527960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7010.546721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.006339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7016.176931] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7022.573111] LustreError: 239407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7022.595815] LustreError: 239407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7024.175425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.308111] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7024.331575] Lustre: 239395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7024.342437] Lustre: Permanently deactivating lustre-MDT0001 [ 7024.354364] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7024.627495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7029.191276] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7029.778306] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7037.195484] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7043.809262] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7051.341560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7051.756128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7057.914311] LustreError: 241248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7057.927585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7058.336411] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7063.017676] LustreError: 241248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7065.733713] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7068.133911] LustreError: 241250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.352944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7072.553682] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7077.872436] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7077.879862] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7078.816517] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7085.816417] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7089.617397] Lustre: Permanently reactivating lustre-MDT0001 [ 7089.635350] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7089.639873] Lustre: Skipped 2 previous similar messages [ 7091.423647] Lustre: lustre-MDT0001: Client 7741ae30-9c8b-4bee-b308-ac848c8904f6 (at 192.168.206.7@tcp) reconnecting [ 7097.076836] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7098.490687] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7098.495513] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.519637] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7098.523995] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7098.540845] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7103.501368] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7103.685465] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7108.008513] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7108.170979] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7110.759374] Lustre: Permanently deactivating lustre-MDT0001 [ 7110.781541] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7110.787072] Lustre: Skipped 2 previous similar messages [ 7116.898106] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7124.965525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.970417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7124.991341] Lustre: Skipped 3 previous similar messages [ 7125.005095] Lustre: Skipped 3 previous similar messages [ 7126.346405] Lustre: server umount lustre-MDT0000 complete [ 7129.701332] LustreError: 239387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392805 with bad export cookie 6955004150741410859 [ 7129.706960] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7129.709917] LustreError: 239387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7129.996786] Lustre: server umount lustre-MDT0001 complete [ 7133.463816] Lustre: server umount lustre-OST0000 complete [ 7143.876441] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:00:18 (1787392818) [ 7153.017379] Lustre: server umount lustre-OST0001 complete [ 7158.984204] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 7168.971418] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 7208.978969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7209.267401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7209.277592] Lustre: Skipped 2 previous similar messages [ 7209.304948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7209.398901] Lustre: lustre-MDT0000: new disk, initializing [ 7209.468055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7209.491818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7214.290436] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7226.216522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7226.370778] Lustre: 247675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7226.429700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7226.434098] Lustre: Skipped 1 previous similar message [ 7226.505952] Lustre: lustre-MDT0001: new disk, initializing [ 7226.630296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7226.692446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7226.713524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7231.003983] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7241.352302] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7247.565378] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7254.828702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7255.176697] Lustre: lustre-OST0000: new disk, initializing [ 7255.182796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7255.189192] Lustre: 249644:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7255.307393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7260.710102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7260.728826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7260.832603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7262.003890] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7272.201332] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7274.441651] LustreError: 247684:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7283.952328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7284.213045] Lustre: lustre-OST0001: new disk, initializing [ 7284.220860] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7284.230962] Lustre: 250763:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7284.544123] LustreError: 247684:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7284.620304] LustreError: 250464:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7290.867437] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7292.117911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7292.144422] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7292.274055] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7294.632094] LustreError: 250464:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7300.572272] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7302.569602] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7307.759554] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7307.761059] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7307.771742] LustreError: Skipped 1 previous similar message [ 7307.790594] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7307.808687] Lustre: Skipped 1 previous similar message [ 7310.961687] Lustre: server umount lustre-OST0001 complete [ 7312.871663] LustreError: 249628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7312.891866] LustreError: 249628:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7321.060979] LustreError: 249630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7321.089578] LustreError: 249630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7331.304994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7331.313582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7331.327219] Lustre: Skipped 2 previous similar messages [ 7331.339166] Lustre: Skipped 1 previous similar message [ 7334.527799] Lustre: server umount lustre-OST0000 complete [ 7339.492893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7339.508549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7339.525787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7344.639532] Lustre: server umount lustre-MDT0000 complete [ 7346.658455] LustreError: 247688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7346.677091] LustreError: 247688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7347.944325] LustreError: 247668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393024 with bad export cookie 6955004150741413456 [ 7347.947838] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7347.957512] LustreError: 247668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7354.428175] Lustre: server umount lustre-MDT0001 complete [ 7364.481628] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 7367.579370] Key type lgssc unregistered [ 7367.890594] LNet: 252794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7367.896236] LNetError: 252794:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7367.913056] LNet: Removed LNI 192.168.206.107@tcp [ 7368.967180] Key type .llcrypt unregistered [ 7368.968783] Key type ._llcrypt unregistered [ 7410.288482] Key type ._llcrypt registered [ 7410.293062] Key type .llcrypt registered [ 7410.411366] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 7411.565915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7411.593697] alg: No test for adler32 (adler32-zlib) [ 7412.699528] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 7413.016373] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 7414.736153] Key type lgssc registered [ 7415.992430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7423.561843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7423.580078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7424.851220] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7424.888452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7425.030183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7429.516846] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7438.241501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7438.378372] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7438.425807] Lustre: 255003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7438.839308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7444.684393] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7452.919853] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7459.525286] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7466.782463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7466.990554] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7467.262238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7472.827774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7475.338885] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7483.865333] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7539.680960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7539.688377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.692603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7539.698012] Lustre: Skipped 1 previous similar message [ 7539.715196] Lustre: Skipped 1 previous similar message [ 7544.105501] Lustre: server umount lustre-OST0000 complete [ 7548.284413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7560.721294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7561.135700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.577598] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7575.041153] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7579.503444] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7579.520266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7579.529822] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7579.547078] Lustre: Skipped 1 previous similar message [ 7615.458970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7615.476155] LustreError: Skipped 1 previous similar message [ 7615.491172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7615.513252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7620.585074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7620.610777] Lustre: Skipped 2 previous similar messages [ 7621.202394] Lustre: server umount lustre-OST0000 complete [ 7625.698357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7625.712150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7625.714751] Lustre: Skipped 2 previous similar messages [ 7625.735465] Lustre: Skipped 1 previous similar message [ 7630.817515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7630.833147] Lustre: Skipped 1 previous similar message [ 7631.764529] Lustre: server umount lustre-MDT0000 complete [ 7635.498121] LustreError: 254995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393311 with bad export cookie 2315348385350777108 [ 7635.510673] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7635.513575] LustreError: 254995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7635.943482] Lustre: server umount lustre-MDT0001 complete [ 7651.343696] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 7655.480153] Key type lgssc unregistered [ 7656.021050] LNet: 260428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7656.029862] LNetError: 260428:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7657.063246] LNet: Removed LNI 192.168.206.107@tcp [ 7658.161571] Key type .llcrypt unregistered [ 7658.162745] Key type ._llcrypt unregistered [ 7687.924238] Key type ._llcrypt registered [ 7687.927865] Key type .llcrypt registered [ 7688.045819] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 7688.832982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7688.938346] alg: No test for adler32 (adler32-zlib) [ 7690.254470] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 7690.556666] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 7692.226631] Key type lgssc registered [ 7693.765549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7703.402884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7703.421705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7704.846020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7710.072864] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7714.276747] LustreError: 261849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7714.291100] LustreError: 261849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7719.376713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7719.395294] LustreError: 261848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7719.779568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7726.722262] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7736.518623] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7744.236798] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7751.849705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7752.126634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7759.779449] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7768.582187] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7774.662370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7805.418885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7805.419637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7805.423166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7805.431693] LustreError: Skipped 1 previous similar message [ 7805.443099] Lustre: Skipped 1 previous similar message [ 7806.784318] Lustre: server umount lustre-OST0000 complete [ 7810.530870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7810.556024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7810.573318] Lustre: Skipped 2 previous similar messages [ 7815.651366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7816.808279] Lustre: server umount lustre-MDT0000 complete [ 7820.529426] LustreError: 261828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393496 with bad export cookie 16795722509784909988 [ 7820.539893] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7820.544181] LustreError: 261828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7820.846921] Lustre: server umount lustre-MDT0001 complete [ 7833.227734] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 7835.965578] Key type lgssc unregistered [ 7836.272828] LNet: 266175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7836.279278] LNetError: 266175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7836.299534] LNet: Removed LNI 192.168.206.107@tcp [ 7837.125755] Key type .llcrypt unregistered [ 7837.130612] Key type ._llcrypt unregistered [ 7853.908412] Key type ._llcrypt registered [ 7853.913812] Key type .llcrypt registered [ 7854.004367] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 7854.892622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7854.960408] alg: No test for adler32 (adler32-zlib) [ 7855.938319] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 7856.183916] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 7857.888145] Key type lgssc registered [ 7859.031494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7867.119516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7867.152292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7868.584381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7873.124436] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7878.118833] LustreError: 267398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7878.132483] LustreError: 267398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7882.870558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7883.235481] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7883.355572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7887.620628] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7896.054207] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7902.468682] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7909.148565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7909.419235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7913.546091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7915.176830] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7922.730993] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7939.046773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7939.055046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7939.075353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7939.086395] Lustre: Skipped 1 previous similar message [ 7944.163608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7945.005436] Lustre: server umount lustre-OST0000 complete [ 7949.283409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7949.302173] Lustre: Skipped 1 previous similar message [ 7949.315459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7949.318047] Lustre: Skipped 1 previous similar message [ 7954.402882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7954.409491] Lustre: Skipped 1 previous similar message [ 7963.104149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7964.643031] LustreError: 267394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7964.677750] LustreError: 267394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7965.269380] Lustre: server umount lustre-MDT0000 complete [ 7969.585877] LustreError: 267378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393645 with bad export cookie 2645251590557677168 [ 7969.599747] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7970.153075] Lustre: server umount lustre-MDT0001 complete [ 7983.426547] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 7987.519230] Key type lgssc unregistered [ 7987.877751] LNet: 271081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7987.889706] LNetError: 271081:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7987.905688] LNet: Removed LNI 192.168.206.107@tcp [ 7988.883573] Key type .llcrypt unregistered [ 7988.888337] Key type ._llcrypt unregistered [ 8014.714394] Key type ._llcrypt registered [ 8014.717502] Key type .llcrypt registered [ 8014.852147] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 8016.390038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8016.450714] alg: No test for adler32 (adler32-zlib) [ 8017.589758] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 8017.840245] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 8019.496793] Key type lgssc registered [ 8020.510678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8029.350432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8029.404512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8030.797632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.337122] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8040.933124] LustreError: 272499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8040.969865] LustreError: 272499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8046.053684] LustreError: 272500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8046.929753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8047.555044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8052.033941] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8059.634181] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8065.097855] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8072.066377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8072.180295] LustreError: 274323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8072.363627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8075.458934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8079.312906] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8088.484454] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8116.710573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8116.711575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8116.727456] LustreError: Skipped 1 previous similar message [ 8116.733232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8116.738959] Lustre: Skipped 1 previous similar message [ 8116.758133] Lustre: Skipped 1 previous similar message [ 8119.514282] Lustre: server umount lustre-OST0000 complete [ 8124.385344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8124.391867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8124.404490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8126.945264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8126.951706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8126.967405] Lustre: Skipped 1 previous similar message [ 8129.246778] Lustre: server umount lustre-MDT0000 complete [ 8132.070186] LustreError: 272500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8132.095541] LustreError: 272500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8133.189989] LustreError: 272479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393809 with bad export cookie 737917301466992238 [ 8133.216545] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8133.625501] Lustre: server umount lustre-MDT0001 complete [ 8145.338724] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 8148.016504] Key type lgssc unregistered [ 8148.343706] LNet: 276807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8148.354244] LNetError: 276807:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8148.389682] LNet: Removed LNI 192.168.206.107@tcp [ 8149.418841] Key type .llcrypt unregistered [ 8149.425177] Key type ._llcrypt unregistered [ 8169.337867] Key type ._llcrypt registered [ 8169.340241] Key type .llcrypt registered [ 8169.458409] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 8170.818433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8170.859146] alg: No test for adler32 (adler32-zlib) [ 8171.897844] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 8172.184246] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 8173.856274] Key type lgssc registered [ 8174.878354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8181.693831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8181.709179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8183.368655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8188.710306] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8193.509881] LustreError: 278059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8193.523748] LustreError: 278059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8197.668984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8197.987217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8202.069561] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8211.059562] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8216.322838] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8223.290273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8223.640693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8228.799713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8229.433808] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8237.810136] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8254.445679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8254.459466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8254.474058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8257.505538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8257.514363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8257.515479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8257.560919] Lustre: Skipped 1 previous similar message [ 8258.835901] Lustre: server umount lustre-OST0000 complete [ 8264.674021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8264.681339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8264.692460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8267.747545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8267.750525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8269.144912] Lustre: server umount lustre-MDT0000 complete [ 8272.874410] LustreError: 278058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8272.909884] LustreError: 278058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8273.628417] LustreError: 278026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393949 with bad export cookie 6111393691901807816 [ 8273.636961] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8273.637308] LustreError: 278026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8274.135580] Lustre: server umount lustre-MDT0001 complete [ 8285.173471] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 8287.986376] Key type lgssc unregistered [ 8288.310802] LNet: 281761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8288.321645] LNetError: 281761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8288.344090] LNet: Removed LNI 192.168.206.107@tcp [ 8289.265578] Key type .llcrypt unregistered [ 8289.270861] Key type ._llcrypt unregistered [ 8301.330466] Key type ._llcrypt registered [ 8301.334095] Key type .llcrypt registered [ 8301.423839] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8320.727731] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 8336.973805] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 8338.123598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8338.163283] alg: No test for adler32 (adler32-zlib) [ 8339.326312] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 8339.609091] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 8341.256936] Key type lgssc registered [ 8342.617314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8379.413839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8379.441563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.713926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8380.773605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8380.917070] Lustre: lustre-MDT0000: new disk, initializing [ 8381.005853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8381.042836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8385.290575] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8396.231501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8396.377990] Lustre: 285318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8396.434746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8396.439919] Lustre: Skipped 1 previous similar message [ 8396.553308] Lustre: lustre-MDT0001: new disk, initializing [ 8396.663816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8396.696493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8396.716788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8400.876794] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8409.588221] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8414.899563] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8420.566620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8420.911515] Lustre: lustre-OST0000: new disk, initializing [ 8420.917332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8420.933082] Lustre: 287287:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8421.051961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8429.245393] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8429.710129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8429.716498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8429.849087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8440.952538] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8445.927195] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8446.209702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8450.591854] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8450.782955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8455.654111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8455.656269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8455.664892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8455.671621] Lustre: Skipped 1 previous similar message [ 8455.689368] Lustre: Skipped 1 previous similar message [ 8458.782449] Lustre: server umount lustre-OST0000 complete [ 8463.330540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8463.346660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8463.367220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8465.895146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8465.896575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8469.222687] Lustre: server umount lustre-MDT0000 complete [ 8472.444293] LustreError: 285310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394148 with bad export cookie 9247414114124586704 [ 8472.451417] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8472.456745] LustreError: 285310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8472.752853] Lustre: server umount lustre-MDT0001 complete [ 8481.829168] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:22:36 (1787394156) [ 8490.037092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.619158] LustreError: 289697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8490.641592] LustreError: 289697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8490.742669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8495.609387] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8496.115395] LustreError: 289698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8500.225138] LustreError: 289697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8505.318502] LustreError: 289698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8506.365506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8506.928807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8511.540803] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8519.509958] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8525.727492] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8532.832990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8533.160820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8539.556982] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8547.502301] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8561.121390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8561.138905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8561.157345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8561.165794] Lustre: Skipped 1 previous similar message [ 8566.510511] Lustre: server umount lustre-OST0000 complete [ 8572.388467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8572.401890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8572.415714] Lustre: Skipped 2 previous similar messages [ 8572.430417] Lustre: Skipped 3 previous similar messages [ 8577.058925] Lustre: server umount lustre-MDT0000 complete [ 8577.514427] LustreError: 289693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8577.549075] LustreError: 289693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8580.680803] LustreError: 289677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394256 with bad export cookie 9247414114124587586 [ 8580.687992] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8580.693809] LustreError: 289677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8581.037475] Lustre: server umount lustre-MDT0001 complete [ 8593.047536] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 8596.626720] Key type lgssc unregistered [ 8597.077853] LNet: 293258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8597.090575] LNetError: 293258:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8597.105066] LNet: Removed LNI 192.168.206.107@tcp [ 8598.018848] Key type .llcrypt unregistered [ 8598.021635] Key type ._llcrypt unregistered [ 8624.983240] Key type ._llcrypt registered [ 8624.988070] Key type .llcrypt registered [ 8625.152969] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 8642.337798] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 8643.157703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8643.178918] alg: No test for adler32 (adler32-zlib) [ 8644.253181] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 8644.558316] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 8646.256230] Key type lgssc registered [ 8647.445354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8695.430385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8695.489165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8696.870434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8696.971584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8697.141515] Lustre: lustre-MDT0000: new disk, initializing [ 8697.285247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8697.306768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8702.582614] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8714.733313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8714.862807] Lustre: 297087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8714.926570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8714.933483] Lustre: Skipped 1 previous similar message [ 8715.067315] Lustre: lustre-MDT0001: new disk, initializing [ 8715.179335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8715.215837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8715.229320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8720.218326] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8729.634945] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8735.486758] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8742.093710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8742.317989] Lustre: lustre-OST2710: new disk, initializing [ 8742.321258] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8742.326820] Lustre: 299056:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8742.444913] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8748.414225] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8748.620841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8748.637105] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8748.772678] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8758.254350] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8765.643146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8765.767981] Lustre: lustre-OST03e8: new disk, initializing [ 8765.771361] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8765.776971] Lustre: 300122:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8765.870668] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8772.249708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8772.271242] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8772.420166] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8772.454735] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8780.815929] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8787.783898] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8788.071876] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8793.499627] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8793.714778] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8798.795676] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8799.045487] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8803.711428] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8803.948804] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8809.367633] Lustre: server umount lustre-MDT0000 complete [ 8812.515209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8812.538418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8813.317560] LustreError: 297080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394489 with bad export cookie 8845527493973980421 [ 8813.328857] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8813.334038] LustreError: 297080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8813.543240] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8813.824614] Lustre: server umount lustre-MDT0001 complete [ 8820.445353] Lustre: server umount lustre-OST2710 complete [ 8824.364637] Lustre: server umount lustre-OST03e8 complete [ 8844.542973] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 8854.146735] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 8894.687252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8894.884052] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8894.911506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8894.977172] Lustre: lustre-MDT0000: new disk, initializing [ 8895.069126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8895.085266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8900.674416] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8912.197646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8912.319211] Lustre: 305368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8912.363565] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8912.367814] Lustre: Skipped 1 previous similar message [ 8912.446413] Lustre: lustre-MDT0001: new disk, initializing [ 8912.541348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8912.573339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8912.589739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8917.093120] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8927.317394] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8932.892976] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8939.172447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8939.455601] Lustre: lustre-OST0000: new disk, initializing [ 8939.459185] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8939.469734] Lustre: 307339:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8939.571257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8945.622910] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8948.268611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8948.283864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8948.372464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8954.858304] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8959.397200] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8959.681653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8964.173747] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8964.395909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8968.675064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8968.678039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8968.699287] Lustre: Skipped 4 previous similar messages [ 8968.718414] Lustre: Skipped 1 previous similar message [ 8972.411470] Lustre: server umount lustre-OST0000 complete [ 8978.921938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8978.929820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8984.033593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8984.042228] Lustre: Skipped 2 previous similar messages [ 8988.130062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8988.142796] Lustre: Skipped 1 previous similar message [ 8990.176134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8990.505339] Lustre: server umount lustre-MDT0000 complete [ 8993.252328] LustreError: 305377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8993.268700] LustreError: 305377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 8994.595636] LustreError: 305362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394670 with bad export cookie 8845527493973982108 [ 8994.607538] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8994.611325] LustreError: 305362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8995.149251] Lustre: server umount lustre-MDT0001 complete [ 9004.310261] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 06:31:18 (1787394678) [ 9005.939557] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9007.468673] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 06:31:22 (1787394682) [ 9010.768669] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 9037.050164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9037.285892] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9037.315892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9037.543732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9042.049416] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9051.355300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9051.535206] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9051.609774] Lustre: 311195:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9057.691225] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9080.508462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9080.717423] Lustre: Denying initial registration attempt from nid 192.168.206.107@tcp, specified as failover [ 9080.728930] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9080.733852] LustreError: 312652:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9080.745582] LustreError: 312652:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9080.749654] LustreError: 312652:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9080.798752] Lustre: server umount lustre-OST0000 complete [ 9080.801641] LustreError: 312652:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9087.976983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9087.986821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9088.008728] Lustre: Skipped 1 previous similar message [ 9088.017779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9098.721193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9098.734284] Lustre: Skipped 5 previous similar messages [ 9102.304149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9102.475154] Lustre: server umount lustre-MDT0000 complete [ 9103.841735] LustreError: 311208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9103.857117] LustreError: 311208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9105.810324] LustreError: 311190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394781 with bad export cookie 8845527493973982990 [ 9105.817363] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9105.820348] LustreError: 311190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9106.060447] Lustre: server umount lustre-MDT0001 complete [ 9115.328739] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 9118.207151] Key type lgssc unregistered [ 9118.484561] LNet: 313773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9118.490752] LNetError: 313773:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9118.512215] LNet: Removed LNI 192.168.206.107@tcp [ 9119.492976] Key type .llcrypt unregistered [ 9119.496533] Key type ._llcrypt unregistered [ 9132.250974] Key type ._llcrypt registered [ 9132.256075] Key type .llcrypt registered [ 9132.407245] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 9133.349571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9133.435349] alg: No test for adler32 (adler32-zlib) [ 9134.659525] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 9134.881969] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 9136.522321] Key type lgssc registered [ 9137.504676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9187.277978] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 9198.476732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9198.510443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9199.807707] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9199.828321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9199.981787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9204.539995] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9214.382834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9214.637304] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9214.741129] Lustre: 316542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9214.993429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9219.902741] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9228.357739] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9235.012405] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9255.726763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9255.963754] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9256.236287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9262.712719] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9270.977440] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9277.409511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9277.415525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9277.423840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9277.928401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9277.933915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9277.962102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9279.414138] Lustre: server umount lustre-OST0000 complete [ 9286.625891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9286.644262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9286.686310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9293.284217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9293.300660] Lustre: Skipped 3 previous similar messages [ 9298.400278] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9298.401629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9298.423411] Lustre: Skipped 1 previous similar message [ 9298.645255] Lustre: server umount lustre-MDT0000 complete [ 9302.675588] LustreError: 317281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394978 with bad export cookie 12340844474434231413 [ 9302.685561] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9302.693976] LustreError: 317281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9302.964952] Lustre: server umount lustre-MDT0001 complete [ 9315.576233] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 9318.562896] Key type lgssc unregistered [ 9318.888516] LNet: 320114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9318.900705] LNetError: 320114:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9319.984568] LNet: Removed LNI 192.168.206.107@tcp [ 9320.927845] Key type .llcrypt unregistered [ 9320.930721] Key type ._llcrypt unregistered [ 9347.749144] Key type ._llcrypt registered [ 9347.756196] Key type .llcrypt registered [ 9347.896277] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 9348.882173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9349.011238] alg: No test for adler32 (adler32-zlib) [ 9350.099624] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 9350.330215] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 9352.008661] Key type lgssc registered [ 9353.107722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9362.165687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9362.225912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9363.778038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9368.782635] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9373.156208] LustreError: 321548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9373.165282] LustreError: 321548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9377.915672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9378.274633] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9378.586776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9383.753659] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9392.493738] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9398.210791] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9407.281809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9407.615867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9413.881086] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9421.317751] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9433.581347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9433.583256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9433.598089] Lustre: Skipped 2 previous similar messages [ 9433.625887] Lustre: Skipped 2 previous similar messages [ 9437.141494] Lustre: server umount lustre-MDT0000 complete [ 9438.707559] LustreError: 321547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9438.734217] LustreError: 321547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9441.212752] LustreError: 321529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395117 with bad export cookie 18133965846241957985 [ 9441.218455] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9441.227529] LustreError: 321529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9443.809524] LustreError: 321542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9443.822557] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9443.842135] LustreError: 321542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9443.858896] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9443.858904] Lustre: Skipped 1 previous similar message [ 9446.818216] LustreError: 323447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9446.818737] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9446.839919] LustreError: 323447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9446.843888] Lustre: Skipped 1 previous similar message [ 9447.711083] Lustre: server umount lustre-MDT0001 complete [ 9454.590719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9465.031695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9466.337760] LustreError: 325128:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9466.349430] LustreError: 325128:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000008f19adb5 x1874219526821248/t0(0) o250->MGC192.168.206.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787395142 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9466.372141] LustreError: 325128:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9466.398326] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfba8c9dbfa517587 [ 9466.424285] Lustre: MGC192.168.206.107@tcp: Connection restored to 0@lo (at 0@lo) [ 9466.954313] LustreError: 323381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9467.089437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9467.146854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9473.678210] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9476.327758] LustreError: 323381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9476.330566] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9476.339360] LustreError: 323381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9484.231867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9484.799236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9489.907384] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9490.368304] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9499.492366] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9501.546376] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9508.052875] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9509.998422] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9518.562225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9518.576203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9518.602771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9520.611716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9520.635173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9530.336670] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9530.685288] Lustre: server umount lustre-OST0000 complete [ 9535.971369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9535.979949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9535.991944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9536.012469] Lustre: Skipped 1 previous similar message [ 9536.031619] Lustre: Skipped 5 previous similar messages [ 9541.593605] Lustre: server umount lustre-MDT0000 complete [ 9546.209916] LustreError: 325147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9546.240408] LustreError: 325147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9546.457117] LustreError: 321529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395222 with bad export cookie 18133965846241965447 [ 9546.461261] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9546.466791] LustreError: 321529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9546.979731] Lustre: server umount lustre-MDT0001 complete [ 9559.912902] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 9563.050815] Key type lgssc unregistered [ 9563.336595] LNet: 327821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9563.351043] LNetError: 327821:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9563.375125] LNet: Removed LNI 192.168.206.107@tcp [ 9564.424910] Key type .llcrypt unregistered [ 9564.426747] Key type ._llcrypt unregistered [ 9592.891802] Key type ._llcrypt registered [ 9592.894231] Key type .llcrypt registered [ 9592.985739] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 9594.338698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9594.418853] alg: No test for adler32 (adler32-zlib) [ 9595.577088] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 9595.916145] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 9597.640252] Key type lgssc registered [ 9598.835762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9608.295555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9608.321957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9609.969209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9615.423247] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9619.433202] LustreError: 329240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9619.472047] LustreError: 329240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9624.507301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9624.560730] LustreError: 329241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9624.892849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9630.036610] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9639.425137] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9646.267505] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9650.660151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9650.674715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9650.695696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9654.756315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9654.760667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9656.675512] Lustre: server umount lustre-MDT0000 complete [ 9659.883209] LustreError: 329240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9659.898752] LustreError: 329240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9661.073171] LustreError: 329220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395337 with bad export cookie 15672461196442931238 [ 9661.101722] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9661.474753] Lustre: server umount lustre-MDT0001 complete [ 9677.688978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9678.024963] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9678.066462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9678.320707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9682.695988] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9693.076794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9693.320246] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9693.390839] Lustre: 331830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9693.832917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9698.747388] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9707.074119] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9714.142458] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9725.364354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9725.677975] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9725.691046] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9725.709679] LustreError: 333715:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9725.725714] LustreError: 333715:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9725.732366] LustreError: 333715:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9725.828441] Lustre: server umount lustre-OST0000 complete [ 9725.833604] LustreError: 333715:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9737.776772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9738.081867] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9738.299477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9745.865649] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9747.082185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9754.298351] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9763.357756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9763.467333] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9763.557360] Lustre: lustre-OST0001: new disk, initializing [ 9763.564586] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9763.585724] Lustre: 335257:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9763.769312] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9770.643743] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9773.718833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9773.737780] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9773.871695] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9779.794379] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9789.410087] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9789.413432] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9789.426802] LustreError: Skipped 1 previous similar message [ 9789.427652] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9789.427656] Lustre: Skipped 1 previous similar message [ 9789.464824] Lustre: Skipped 1 previous similar message [ 9790.661523] Lustre: server umount lustre-OST0001 complete [ 9794.543983] LustreError: 334181:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9794.596106] LustreError: 334181:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9798.113733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9798.125831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9798.140446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9798.148235] Lustre: Skipped 1 previous similar message [ 9799.665402] LustreError: 334182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9799.693034] LustreError: 334182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9804.774703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9804.788167] Lustre: Skipped 4 previous similar messages [ 9809.900104] LustreError: 334183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9809.910424] LustreError: 334183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9810.400168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9810.825681] Lustre: server umount lustre-OST0000 complete [ 9815.021498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9815.032936] Lustre: Skipped 1 previous similar message [ 9815.038088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9815.047487] Lustre: Skipped 2 previous similar messages [ 9816.552305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9816.564482] LustreError: Skipped 1 previous similar message [ 9829.345218] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9829.642913] Lustre: server umount lustre-MDT0000 complete [ 9830.368690] LustreError: 332609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9830.393288] LustreError: 332609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9833.704287] LustreError: 334198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395509 with bad export cookie 15672461196442931791 [ 9833.707817] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9833.722113] LustreError: 334198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9834.064057] Lustre: server umount lustre-MDT0001 complete [ 9846.449246] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 9849.414370] Key type lgssc unregistered [ 9849.836820] LNet: 337218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9849.846838] LNetError: 337218:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9849.877570] LNet: Removed LNI 192.168.206.107@tcp [ 9850.940944] Key type .llcrypt unregistered [ 9850.946646] Key type ._llcrypt unregistered [ 9928.378325] Key type ._llcrypt registered [ 9928.380777] Key type .llcrypt registered [ 9928.476092] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 9942.845653] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 9944.289348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9944.351991] alg: No test for adler32 (adler32-zlib) [ 9945.481247] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 9945.733186] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 9947.432159] Key type lgssc registered [ 9948.522307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9990.794254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9990.828881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9992.194210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9992.237486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9992.329815] Lustre: lustre-MDT0000: new disk, initializing [ 9992.501120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9992.526357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9997.701339] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10008.151865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10008.277951] Lustre: 342157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10008.337781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10008.341220] Lustre: Skipped 1 previous similar message [10008.476974] Lustre: lustre-MDT0001: new disk, initializing [10008.630255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10008.716804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10008.724805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10013.514317] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10022.697376] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10029.172176] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10035.944680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10036.279234] Lustre: lustre-OST0000: new disk, initializing [10036.282493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10036.288048] Lustre: 344122:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10036.356907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10042.739583] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10043.984444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10043.993941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10044.080224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10052.608614] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10057.444747] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10057.667410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10062.268535] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10062.533447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10064.867268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10064.876860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10064.882798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10064.888990] Lustre: Skipped 1 previous similar message [10064.907958] Lustre: Skipped 1 previous similar message [10069.986525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10069.999559] Lustre: Skipped 1 previous similar message [10071.099365] Lustre: server umount lustre-OST0000 complete [10080.228235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10080.233172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10080.244639] LustreError: Skipped 1 previous similar message [10080.250634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10080.265368] Lustre: Skipped 1 previous similar message [10080.297367] Lustre: Skipped 1 previous similar message [10081.957241] Lustre: server umount lustre-MDT0000 complete [10086.441048] LustreError: 342149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395762 with bad export cookie 8440685802933339827 [10086.447989] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10086.455867] LustreError: 342149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10086.765741] Lustre: server umount lustre-MDT0001 complete [10096.819178] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 06:49:31 (1787395771) [10105.154185] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 06:49:39 (1787395779) [10113.822538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10114.287576] LustreError: 346879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.309372] LustreError: 346879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10114.380205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10119.586765] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10119.650341] LustreError: 346880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10124.773289] LustreError: 346879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.902220] LustreError: 346880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10130.589173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10131.056188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10136.238995] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10144.310991] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10150.391065] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10157.811213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10158.104871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10164.459292] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10172.256142] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10182.985891] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10184.749195] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10186.258234] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10187.917730] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10189.864251] Lustre: Failing over lustre-MDT0000 [10190.132294] Lustre: server umount lustre-MDT0000 complete [10190.817798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10190.821426] LustreError: 346874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10190.842779] Lustre: Skipped 2 previous similar messages [10190.884128] LustreError: 346874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10199.502393] LustreError: 347990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10199.529061] LustreError: 347990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10206.050841] Lustre: 340377:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395866/real 1787395866] req@000000008195d5e0 x1874220150759296/t0(0) o400->MGC192.168.206.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787395882 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10206.099767] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10209.164927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10216.425703] LustreError: 340376:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a041c31d x1874220150764160/t0(0) o250->MGC192.168.206.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10216.457306] LustreError: 346879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10216.488370] LustreError: 346879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [10216.826530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10216.924681] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10217.972280] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10222.067436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10222.095862] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10222.175771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10222.511782] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10232.410141] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10234.288554] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10238.274558] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10240.919824] Lustre: Failing over lustre-MDT0000 [10241.351664] Lustre: server umount lustre-MDT0000 complete [10242.532432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10242.539497] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10242.570888] Lustre: Skipped 1 previous similar message [10248.650524] LustreError: 346876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10248.670862] LustreError: 346876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10258.912837] Lustre: 340379:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395918/real 1787395918] req@000000000a4d074c x1874220150779904/t0(0) o400->MGC192.168.206.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787395934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10258.953426] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10261.065850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10269.155083] LustreError: 340376:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000584d835d x1874220150784768/t0(0) o250->MGC192.168.206.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10269.571721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10269.671990] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10270.156616] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10274.803540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10274.826047] Lustre: Skipped 2 previous similar messages [10274.863284] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10274.920593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10275.407035] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10286.051426] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10287.789620] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10295.269614] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10295.280205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10295.306677] Lustre: Skipped 1 previous similar message [10295.323624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10295.332735] Lustre: Skipped 2 previous similar messages [10300.283354] Lustre: server umount lustre-MDT0000 complete [10304.620176] LustreError: 346859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395980 with bad export cookie 8440685802933342536 [10304.657328] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10305.192901] Lustre: server umount lustre-MDT0001 complete [10320.215225] Lustre: server umount lustre-OST0000 complete [10337.268764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10337.612740] LustreError: 353224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10337.632829] LustreError: 353224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [10337.683398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10341.977906] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10350.374410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10355.670464] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10364.077730] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10369.519674] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10376.048275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10382.566299] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10382.643118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10389.749436] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10406.369601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10406.376456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10406.387267] Lustre: Skipped 2 previous similar messages [10406.405582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10406.416135] Lustre: Skipped 1 previous similar message [10411.794534] Lustre: server umount lustre-OST0000 complete [10416.610408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10416.619168] Lustre: Skipped 3 previous similar messages [10422.319560] Lustre: server umount lustre-MDT0000 complete [10426.399964] LustreError: 353204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396102 with bad export cookie 8440685802933343285 [10426.414968] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10426.424816] LustreError: 353204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10426.830293] Lustre: server umount lustre-MDT0001 complete [10439.231214] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [10442.162298] Key type lgssc unregistered [10442.514815] LNet: 356778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10442.521841] LNetError: 356778:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10442.539942] LNet: Removed LNI 192.168.206.107@tcp [10443.374326] Key type .llcrypt unregistered [10443.376167] Key type ._llcrypt unregistered [10470.151315] Key type ._llcrypt registered [10470.153645] Key type .llcrypt registered [10470.269652] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [10471.600936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10471.649202] alg: No test for adler32 (adler32-zlib) [10472.748616] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [10473.018060] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [10474.704174] Key type lgssc registered [10475.890321] Lustre: Echo OBD driver; http://www.lustre.org/ [10483.874939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10483.888561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10485.260808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10490.435194] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10495.464461] LustreError: 358293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10495.497988] LustreError: 358293:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10499.628277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10499.876960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10503.889353] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10511.221285] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10516.787658] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10524.752407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10525.148852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10530.300067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10531.614280] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10539.917825] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10556.000577] Lustre: server umount lustre-MDT0000 complete [10556.395953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10556.407551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10556.425580] LustreError: 359343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10559.893513] LustreError: 358273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396235 with bad export cookie 14951081859032765926 [10559.918511] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10559.970114] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10559.974076] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10559.987571] Lustre: Skipped 1 previous similar message [10559.990155] LustreError: 361108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10560.374987] Lustre: server umount lustre-MDT0001 complete [10566.578996] Lustre: server umount lustre-OST0000 complete [10580.606889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10581.143430] LustreError: 362037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10581.264702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10586.605263] LustreError: 362038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10587.038103] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10596.249993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10596.695975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10601.702196] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10610.787992] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10617.220687] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10624.155546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10624.537226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10626.625445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10630.743761] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10638.923445] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10652.134437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10652.143384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10652.156435] Lustre: Skipped 1 previous similar message [10652.178679] Lustre: Skipped 2 previous similar messages [10657.258524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10657.271821] Lustre: Skipped 2 previous similar messages [10657.606502] Lustre: server umount lustre-MDT0000 complete [10661.530093] LustreError: 362017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396337 with bad export cookie 14951081859032767053 [10661.531798] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10661.545224] LustreError: 362017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10661.808861] Lustre: server umount lustre-MDT0001 complete [10675.381025] Lustre: server umount lustre-OST0000 complete [10691.767194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10692.352707] LustreError: 365948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10692.381523] LustreError: 365948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10692.446659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10696.818115] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10704.711460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10709.402423] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10717.275159] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10723.006425] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10729.754461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10730.073140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10730.078504] Lustre: Skipped 1 previous similar message [10736.356849] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10737.297560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10745.081959] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10751.974655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10751.985647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10752.002726] Lustre: Skipped 1 previous similar message [10752.016606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10757.576863] Lustre: server umount lustre-OST0000 complete [10761.700421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10761.707192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10761.715413] Lustre: Skipped 1 previous similar message [10761.720721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10761.728145] Lustre: Skipped 2 previous similar messages [10767.263256] Lustre: server umount lustre-MDT0000 complete [10768.365083] LustreError: 365950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10768.392858] LustreError: 365950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10771.036452] LustreError: 365929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396447 with bad export cookie 14951081859032768075 [10771.048487] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10771.434916] Lustre: server umount lustre-MDT0001 complete [10783.647237] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [10787.238735] Key type lgssc unregistered [10787.596698] LNet: 369493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10787.601813] LNetError: 369493:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10787.626722] LNet: Removed LNI 192.168.206.107@tcp [10788.488520] Key type .llcrypt unregistered [10788.490350] Key type ._llcrypt unregistered [10819.055391] Key type ._llcrypt registered [10819.058749] Key type .llcrypt registered [10819.259625] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [10820.734755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10821.096365] alg: No test for adler32 (adler32-zlib) [10822.467872] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [10822.732606] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [10824.416158] Key type lgssc registered [10826.195372] Lustre: Echo OBD driver; http://www.lustre.org/ [10834.655752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10834.671770] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10834.678760] LustreError: 370934:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10834.697785] LustreError: 370934:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10834.707806] LustreError: 370934:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10834.721151] LustreError: 370934:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10834.727461] LustreError: 370934:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10841.997116] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10842.004635] LustreError: 371331:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10842.019843] LustreError: 371331:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10842.024889] LustreError: 371331:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10842.030641] LustreError: 371331:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10842.036813] LustreError: 371331:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10864.239310] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [10867.042885] Key type lgssc unregistered [10867.406605] LNet: 372337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10867.412466] LNetError: 372337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10867.433404] LNet: Removed LNI 192.168.206.107@tcp [10868.561679] Key type .llcrypt unregistered [10868.564624] Key type ._llcrypt unregistered [10888.623269] Key type ._llcrypt registered [10888.630518] Key type .llcrypt registered [10888.752972] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [10903.911240] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [10904.843932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10904.914883] alg: No test for adler32 (adler32-zlib) [10905.993275] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [10906.222476] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [10907.915434] Key type lgssc registered [10909.096324] Lustre: Echo OBD driver; http://www.lustre.org/ [10943.223555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10943.247927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10944.509817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10944.567120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10944.665833] Lustre: lustre-MDT0000: new disk, initializing [10944.772910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10944.811169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10949.832945] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10960.901518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10961.027979] Lustre: 375495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10961.123156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10961.127110] Lustre: Skipped 1 previous similar message [10961.243367] Lustre: lustre-MDT0001: new disk, initializing [10961.371836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10961.451113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10961.465487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10966.344726] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10976.479478] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10983.248629] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10990.590968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10990.977884] Lustre: lustre-OST0000: new disk, initializing [10990.981946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10990.989177] Lustre: 377462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10991.087216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10997.513255] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10999.362302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10999.387048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10999.530405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11007.340952] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11012.337464] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11012.502943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11017.372153] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11017.641904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11020.259073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11020.262488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11020.277996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11020.289554] Lustre: Skipped 1 previous similar message [11025.379973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11025.385871] Lustre: Skipped 1 previous similar message [11025.549504] Lustre: server umount lustre-OST0000 complete [11030.497835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11030.506708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11030.519582] Lustre: Skipped 1 previous similar message [11030.545067] Lustre: Skipped 2 previous similar messages [11035.618576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11035.628227] Lustre: Skipped 1 previous similar message [11035.968407] Lustre: server umount lustre-MDT0000 complete [11039.581433] LustreError: 375488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396715 with bad export cookie 6536476819620959560 [11039.582299] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11039.590714] LustreError: 375488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11039.988107] Lustre: server umount lustre-MDT0001 complete [11048.498458] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11050.237658] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:05:24 (1787396724) [11057.814936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11058.354136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11063.341977] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11063.777384] LustreError: 379923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11063.798950] LustreError: 379923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11068.901303] LustreError: 379922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11072.280723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11072.581840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11077.456429] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11085.327046] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11090.779423] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11098.569350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11099.051841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11107.842362] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11118.150421] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11125.488858] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11125.716823] Lustre: lustre-OST0001: new disk, initializing [11125.719936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11125.742050] Lustre: 382759:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11125.832477] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11131.904094] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11135.620727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11135.639388] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11135.721432] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11142.330069] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11146.216680] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11146.222195] LustreError: Skipped 1 previous similar message [11146.227123] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11146.236759] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11146.240284] Lustre: Skipped 1 previous similar message [11146.248031] Lustre: Skipped 1 previous similar message [11152.308611] Lustre: server umount lustre-OST0001 complete [11154.398401] LustreError: 381745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11156.484098] LustreError: 381746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11156.520097] LustreError: 381746:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11161.571967] LustreError: 383255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11161.589211] LustreError: 383255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11169.771424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11169.789678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11169.821417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11169.830779] Lustre: Skipped 3 previous similar messages [11171.830535] LustreError: 381745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11171.848364] LustreError: 381745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11175.215696] Lustre: server umount lustre-OST0000 complete [11180.005227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11180.010774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11180.024761] Lustre: Skipped 1 previous similar message [11186.008886] Lustre: server umount lustre-MDT0000 complete [11189.707530] LustreError: 379903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396865 with bad export cookie 6536476819620960442 [11189.720665] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11189.727502] LustreError: 379903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11204.064129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11204.316759] Lustre: server umount lustre-MDT0001 complete [11215.193777] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [11217.917847] Key type lgssc unregistered [11218.158651] LNet: 384751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11218.163746] LNetError: 384751:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11218.177702] LNet: Removed LNI 192.168.206.107@tcp [11218.858261] Key type .llcrypt unregistered [11218.860489] Key type ._llcrypt unregistered [11271.512032] Key type ._llcrypt registered [11271.517474] Key type .llcrypt registered [11271.633903] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [11272.928756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11273.028043] alg: No test for adler32 (adler32-zlib) [11274.059957] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [11274.262902] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [11275.920489] Key type lgssc registered [11277.179265] Lustre: Echo OBD driver; http://www.lustre.org/ [11284.386443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11284.423285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11285.709248] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11285.767597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11286.012232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11291.058251] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11300.703926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11300.900961] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11301.004443] Lustre: 387549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11301.480332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11305.818682] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11313.993664] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11320.292982] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11326.945427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11326.957599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11326.975592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11327.459345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11328.841391] Lustre: server umount lustre-MDT0000 complete [11332.332102] LustreError: 387541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397008 with bad export cookie 17452450345229039544 [11332.344289] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11332.600688] Lustre: server umount lustre-MDT0001 complete [11341.492383] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 07:10:16 (1787397016) [11348.381022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11348.831559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11353.686838] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11354.083768] LustreError: 390061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11354.113370] LustreError: 390061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11359.203995] LustreError: 390060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11361.905052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11362.296564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11367.376725] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11375.335450] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11381.387304] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11388.432332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11388.658546] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11388.982183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11394.842813] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11403.761897] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11413.823985] Lustre: Permanently deactivating lustre-OST0000 [11413.864636] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11420.739632] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11422.764866] LustreError: 393000:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11422.780268] LustreError: 393000:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11422.788565] LustreError: 393000:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11426.102313] LustreError: 393049:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11426.111405] LustreError: 393049:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11434.515861] Lustre: server umount lustre-OST0000 complete [11437.120094] LustreError: 393298:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11437.128858] LustreError: 393298:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11439.586619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11439.603912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11439.609575] Lustre: Skipped 2 previous similar messages [11444.194683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11444.200270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11444.213846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11449.832932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11449.847236] Lustre: Skipped 2 previous similar messages [11449.935659] Lustre: server umount lustre-MDT0000 complete [11453.914237] LustreError: 390040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397129 with bad export cookie 17452450345229040118 [11453.924323] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11453.935283] LustreError: 390040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11464.397336] Lustre: server umount lustre-MDT0001 complete [11475.868679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11481.341894] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11484.688146] LustreError: 394460:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11493.025159] Lustre: 394799:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.107@tcp [11495.951515] Lustre: server umount MGS complete [11509.151210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11509.951246] LustreError: 395524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11509.998935] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11510.010174] Lustre: Skipped 1 previous similar message [11510.107067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11515.363448] LustreError: 395524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11515.389060] LustreError: 395524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11515.518964] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11520.483655] LustreError: 395525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11525.653231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11525.889296] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11531.364061] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11539.040121] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11545.692842] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11547.692720] Lustre: Permanently reactivating lustre-OST0000 [11547.710636] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11547.718304] Lustre: Skipped 2 previous similar messages [11574.240680] LustreError: 397264:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11582.760453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11583.226790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11583.239329] Lustre: Skipped 1 previous similar message [11590.335476] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11598.794517] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11604.961311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11604.975155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11604.998569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11605.010420] Lustre: Skipped 1 previous similar message [11605.030569] Lustre: Skipped 1 previous similar message [11606.775211] Lustre: server umount lustre-OST0000 complete [11613.154573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11613.159769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11613.165798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11616.884635] Lustre: server umount lustre-MDT0000 complete [11620.323314] LustreError: 395519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11620.348498] LustreError: 395519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11621.028542] LustreError: 395504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397297 with bad export cookie 17452450345229041175 [11621.032284] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11621.037384] LustreError: 395504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11621.371961] Lustre: server umount lustre-MDT0001 complete [11634.029178] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [11636.892762] Key type lgssc unregistered [11637.192522] LNet: 399842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11637.203407] LNetError: 399842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11637.221692] LNet: Removed LNI 192.168.206.107@tcp [11638.329689] Key type .llcrypt unregistered [11638.333480] Key type ._llcrypt unregistered [11657.089673] Key type ._llcrypt registered [11657.094283] Key type .llcrypt registered [11657.253142] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [11675.092105] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [11676.141522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11676.181667] alg: No test for adler32 (adler32-zlib) [11677.259650] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [11677.516583] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [11679.296182] Key type lgssc registered [11680.611371] Lustre: Echo OBD driver; http://www.lustre.org/ [11715.673963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11715.699413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11716.964833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11717.004906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11717.093504] Lustre: lustre-MDT0000: new disk, initializing [11717.186440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11717.204565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11721.609917] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11732.249870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11732.360138] Lustre: 403000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11732.425715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11732.432627] Lustre: Skipped 1 previous similar message [11732.495967] Lustre: lustre-MDT0001: new disk, initializing [11732.580610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11732.615485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11732.629104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11737.684132] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11747.521388] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11753.422400] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11760.828044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11761.143767] Lustre: lustre-OST0000: new disk, initializing [11761.155482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11761.165338] Lustre: 404968:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11761.279652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11767.547707] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11771.496877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11771.518047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11771.629123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11777.714747] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11784.181693] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11784.567674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11790.436537] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11790.719513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11795.430757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11795.437752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11795.449710] Lustre: Skipped 1 previous similar message [11795.490255] Lustre: Skipped 1 previous similar message [11800.554284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11800.569671] Lustre: Skipped 1 previous similar message [11805.683139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11805.691856] Lustre: Skipped 1 previous similar message [11807.200187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11807.526902] Lustre: server umount lustre-OST0000 complete [11814.369442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11814.375407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11814.387334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11815.905829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11817.595116] Lustre: server umount lustre-MDT0000 complete [11821.173850] LustreError: 402992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397497 with bad export cookie 11425662397041263668 [11821.178910] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11821.196951] LustreError: 402992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11821.498845] Lustre: server umount lustre-MDT0001 complete [11830.000674] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11831.319912] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 07:18:26 (1787397506) [11837.804311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11838.383604] LustreError: 407429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11838.395173] LustreError: 407429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11838.470194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11842.630490] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11843.554147] LustreError: 407430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11848.677246] LustreError: 407429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11852.826031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11853.254800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11857.405989] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11864.826777] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11871.230519] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11878.493221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11878.823883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11884.888261] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11891.338504] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11893.069163] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11895.312430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11900.401526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11900.408417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11900.420803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11900.424803] Lustre: Skipped 2 previous similar messages [11904.483980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11906.141245] Lustre: server umount lustre-OST0000 complete [11914.721499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11914.723748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11914.739373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11914.751784] Lustre: Skipped 1 previous similar message [11914.762278] Lustre: Skipped 3 previous similar messages [11916.751096] Lustre: server umount lustre-MDT0000 complete [11919.845293] LustreError: 407429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11919.858214] LustreError: 407429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11920.772734] LustreError: 407411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397596 with bad export cookie 11425662397041264550 [11920.780955] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11920.785960] LustreError: 407411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11921.206910] Lustre: server umount lustre-MDT0001 complete [11932.735300] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [11935.210513] Key type lgssc unregistered [11935.451732] LNet: 411156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11935.462363] LNetError: 411156:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11935.482907] LNet: Removed LNI 192.168.206.107@tcp [11936.390990] Key type .llcrypt unregistered [11936.394143] Key type ._llcrypt unregistered [11959.073190] Key type ._llcrypt registered [11959.075420] Key type .llcrypt registered [11959.187690] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [11959.413488] Key type .llcrypt unregistered [11959.415572] Key type ._llcrypt unregistered [11976.873885] Key type ._llcrypt registered [11976.876961] Key type .llcrypt registered [11976.977244] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [11978.048048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11978.081250] alg: No test for adler32 (adler32-zlib) [11979.143498] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [11979.359869] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [11980.993293] Key type lgssc registered [11981.924261] Lustre: Echo OBD driver; http://www.lustre.org/ [11989.389179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11989.409346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11990.856521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11995.146811] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12000.228128] LustreError: 413372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12000.250728] LustreError: 413372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12003.810510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12004.179241] LustreError: 413371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12004.200954] LustreError: 413371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12004.231472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12009.442378] LustreError: 413372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12009.469836] LustreError: 413372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12011.703121] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12014.564242] LustreError: 413367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12014.577808] LustreError: 413367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12015.610962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12018.797638] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12020.716501] LustreError: 413372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12020.737116] LustreError: 413372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12025.180762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12025.510197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12029.927422] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12041.040344] Lustre: server umount lustre-OST0000 complete [12046.308403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12046.314548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12046.324614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12046.340970] Lustre: Skipped 1 previous similar message [12046.351596] Lustre: Skipped 1 previous similar message [12051.428166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12051.444772] Lustre: Skipped 1 previous similar message [12054.584112] Lustre: server umount lustre-MDT0000 complete [12056.552258] LustreError: 413367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12056.564542] LustreError: 413367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12057.568155] Lustre: 412683:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397717/real 1787397717] req@00000000cb05ebbb x1874222283515008/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787397733 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12057.594415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12058.604096] LustreError: 413352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397734 with bad export cookie 13862800310919880333 [12058.612111] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12058.627859] LustreError: 413352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12058.948500] Lustre: server umount lustre-MDT0001 complete [12070.834272] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [12074.285372] Key type lgssc unregistered [12074.659110] LNet: 416517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12074.679297] LNetError: 416517:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12074.701190] LNet: Removed LNI 192.168.206.107@tcp [12075.867553] Key type .llcrypt unregistered [12075.870063] Key type ._llcrypt unregistered [12102.391249] Key type ._llcrypt registered [12102.394714] Key type .llcrypt registered [12102.477128] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [12103.413807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12103.431472] alg: No test for adler32 (adler32-zlib) [12104.442672] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [12104.634326] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [12106.304169] Key type lgssc registered [12107.330898] Lustre: Echo OBD driver; http://www.lustre.org/ [12113.994469] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12114.015706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12129.570178] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12134.688320] LustreError: 417909:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.107@tcp: failed processing log, type 4: rc = -110 [12161.444515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12167.680664] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12175.306775] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12181.672667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12182.309456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12182.391588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12187.258398] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12187.621826] LustreError: 417935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12187.642128] LustreError: 417935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12192.741117] LustreError: 417934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12195.269365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12195.720250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12200.033178] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12208.203096] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12213.740669] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12222.442892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12222.444392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12222.469148] Lustre: Skipped 1 previous similar message [12224.011212] Lustre: server umount lustre-OST0000 complete [12229.093199] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12229.098366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12229.107081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12229.110817] Lustre: Skipped 1 previous similar message [12230.223245] Lustre: server umount lustre-MDT0000 complete [12232.673970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12232.678700] LustreError: 418951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12232.720058] LustreError: 418951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12234.260657] LustreError: 417917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397910 with bad export cookie 8836022345107458260 [12234.274936] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12234.278901] LustreError: 417917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12234.571067] Lustre: server umount lustre-MDT0001 complete [12246.045934] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [12248.776745] Key type lgssc unregistered [12249.086928] LNet: 421479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12249.102367] LNetError: 421479:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12250.156961] LNet: Removed LNI 192.168.206.107@tcp [12251.152550] Key type .llcrypt unregistered [12251.159629] Key type ._llcrypt unregistered [12277.522957] Key type ._llcrypt registered [12277.526248] Key type .llcrypt registered [12277.627829] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [12278.838403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12278.902632] alg: No test for adler32 (adler32-zlib) [12279.940300] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [12280.206433] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [12281.857262] Key type lgssc registered [12283.111516] Lustre: Echo OBD driver; http://www.lustre.org/ [12290.572208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12290.599914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12291.980849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12295.961968] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12302.308305] LustreError: 422899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12302.329213] LustreError: 422899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12305.946830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12306.404937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12310.844580] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12319.467707] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12324.891416] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12331.847249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12331.943872] LustreError: 424721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12332.184262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12337.328131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12337.739309] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12344.240150] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12350.951530] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.7@tcp (stopping) [12352.482589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12352.489777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12352.500714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12352.999605] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12354.295824] Lustre: server umount lustre-MDT0000 complete [12356.042664] LustreError: 422893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12356.538888] LustreError: 422879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.7@tcp arrived at 1787398032 with bad export cookie 2053085477948497751 [12356.553863] LustreError: 422879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12358.118029] LustreError: 422898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12358.140879] LustreError: 422898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12358.276443] LustreError: 422878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398034 with bad export cookie 2053085477948496855 [12358.283547] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12358.296351] LustreError: 422878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12363.235186] LustreError: 422899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12363.238149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12363.247264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12363.247273] Lustre: Skipped 1 previous similar message [12363.247936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12363.247940] Lustre: Skipped 2 previous similar messages [12363.258437] LustreError: 422899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12364.503092] Lustre: server umount lustre-OST0000 complete [12371.497716] Lustre: server umount lustre-MDT0001 complete [12381.877770] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [12384.525606] Key type lgssc unregistered [12384.815912] LNet: 426583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12384.827176] LNetError: 426583:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12384.843258] LNet: Removed LNI 192.168.206.107@tcp [12385.761161] Key type .llcrypt unregistered [12385.763902] Key type ._llcrypt unregistered [12408.447792] Key type ._llcrypt registered [12408.454482] Key type .llcrypt registered [12408.654437] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [12409.669414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12409.798515] alg: No test for adler32 (adler32-zlib) [12410.892914] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [12411.168580] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [12412.840176] Key type lgssc registered [12414.336674] Lustre: Echo OBD driver; http://www.lustre.org/ [12423.128966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12423.155913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12424.921098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12429.898258] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12434.409023] LustreError: 428001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12434.430412] LustreError: 428001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12439.525243] LustreError: 428000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12440.823501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12441.410246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12445.812566] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12453.469289] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12458.747173] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12465.383088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12465.875820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12469.938496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12472.459799] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12480.268833] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12485.606859] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12485.610468] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12485.625671] Lustre: Skipped 1 previous similar message [12485.638956] Lustre: Skipped 1 previous similar message [12487.147690] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.7@tcp (stopping) [12490.100754] Lustre: server umount lustre-MDT0001 complete [12490.725993] LustreError: 428000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12490.741152] LustreError: 428000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12495.843520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12495.851273] LustreError: 428000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12495.851515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12495.863613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12500.966883] LustreError: 428001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12500.970488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12501.803463] Lustre: server umount lustre-OST0000 complete [12506.118876] Lustre: server umount lustre-MDT0000 complete [12518.425690] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [12521.003887] Key type lgssc unregistered [12521.294310] LNet: 431684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12521.304978] LNetError: 431684:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12521.320468] LNet: Removed LNI 192.168.206.107@tcp [12522.011928] Key type .llcrypt unregistered [12522.015099] Key type ._llcrypt unregistered [12546.261576] Key type ._llcrypt registered [12546.264589] Key type .llcrypt registered [12546.375967] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [12560.512581] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [12561.469599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12561.566222] alg: No test for adler32 (adler32-zlib) [12562.686464] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [12563.004620] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [12564.657674] Key type lgssc registered [12565.852128] Lustre: Echo OBD driver; http://www.lustre.org/ [12601.959860] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [12620.377807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12620.407815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12621.628325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12621.670414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12621.781376] Lustre: lustre-MDT0000: new disk, initializing [12621.888465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12621.908922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12626.467246] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12642.189501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12642.570749] Lustre: lustre-OST0000: new disk, initializing [12642.577125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12642.593462] Lustre: Skipped 1 previous similar message [12642.602314] Lustre: 436721:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12642.703570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12649.026545] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12649.570917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12649.578857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12649.618919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12657.944156] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12673.354828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12673.560127] Lustre: 435701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12673.689302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12673.838221] Lustre: lustre-MDT0001: new disk, initializing [12674.018525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12674.057137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12674.079062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12678.830830] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12687.795205] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12687.959420] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12691.964147] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12692.115660] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12697.063624] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12697.072681] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12697.082189] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12697.104272] Lustre: Skipped 1 previous similar message [12702.192440] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12702.204292] Lustre: Skipped 3 previous similar messages [12702.916972] Lustre: server umount lustre-MDT0001 complete [12707.793409] LustreError: 435709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12707.817703] LustreError: 435709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12712.422756] LustreError: 437624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12712.445938] LustreError: 437624:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12717.545342] LustreError: 437624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12717.563743] LustreError: 437624:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12722.658857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12722.660414] LustreError: 435714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12722.686778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12727.777696] LustreError: 436706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12727.806809] LustreError: 436706:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12728.247859] Lustre: server umount lustre-OST0000 complete [12739.560932] Lustre: server umount lustre-MDT0000 complete [12757.988597] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [12761.755732] Key type lgssc unregistered [12761.959060] LNet: 440468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12761.966171] LNetError: 440468:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12762.986402] LNet: Removed LNI 192.168.206.107@tcp [12764.138608] Key type .llcrypt unregistered [12764.140628] Key type ._llcrypt unregistered [12803.205864] Key type ._llcrypt registered [12803.209879] Key type .llcrypt registered [12803.317608] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [12803.600429] Key type .llcrypt unregistered [12803.609667] Key type ._llcrypt unregistered [12821.387918] Key type ._llcrypt registered [12821.390493] Key type .llcrypt registered [12821.491580] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [12822.402641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12822.462201] alg: No test for adler32 (adler32-zlib) [12823.652028] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [12823.920883] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [12825.568197] Key type lgssc registered [12826.470324] Lustre: Echo OBD driver; http://www.lustre.org/ [12851.578550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12851.601943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12852.924313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12852.979742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12853.076642] Lustre: lustre-MDT0000: new disk, initializing [12853.185517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12853.210070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12858.315465] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12871.710631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12871.813864] Lustre: 443845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12871.882993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12871.887213] Lustre: Skipped 1 previous similar message [12871.987626] Lustre: lustre-MDT0001: new disk, initializing [12872.171117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12872.208097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12872.234067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12877.436850] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12886.511357] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12891.650260] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12898.201801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12898.504101] Lustre: lustre-OST0000: new disk, initializing [12898.507800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12898.512465] Lustre: 445813:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12898.620894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12904.157453] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12905.038512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12905.044403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12905.171721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12913.839586] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12920.809558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12920.811223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12920.818933] Lustre: Skipped 2 previous similar messages [12920.828265] Lustre: Skipped 2 previous similar messages [12925.928416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12925.938985] Lustre: Skipped 2 previous similar messages [12926.032608] Lustre: server umount lustre-MDT0000 complete [12929.442436] LustreError: 444651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398605 with bad export cookie 12260959910731405527 [12929.443806] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12929.455238] LustreError: 444651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12929.713261] Lustre: server umount lustre-MDT0001 complete [12943.743109] Lustre: server umount lustre-OST0000 complete [12953.439395] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:37:08 (1787398628) [12961.408229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12961.853801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12966.163907] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12966.883715] LustreError: 447845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12966.896897] LustreError: 447845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12970.981620] LustreError: 447844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12974.629552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12975.121036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12979.902223] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12986.936722] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12992.661019] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13011.176778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13011.338714] Lustre: Found index 0 for lustre-OST0000, updating log [13011.619817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13017.591309] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13025.420952] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13032.119416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13047.794534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13047.807150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13047.807716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13047.816420] Lustre: Skipped 1 previous similar message [13049.729290] Lustre: server umount lustre-OST0000 complete [13064.799370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13064.808531] Lustre: Skipped 2 previous similar messages [13064.812252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13064.829635] Lustre: Skipped 3 previous similar messages [13070.305396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13070.314868] Lustre: Skipped 1 previous similar message [13071.142305] Lustre: server umount lustre-MDT0000 complete [13073.824165] LNet: 1 local NIs in recovery (showing 1): 192.168.206.107@tcp [13074.597730] LustreError: 447825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398750 with bad export cookie 12260959910731406920 [13074.603331] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13074.606145] LustreError: 447825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13075.076092] Lustre: server umount lustre-MDT0001 complete [13083.282718] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 07:39:18 (1787398758) [13703.803250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13704.340059] LustreError: 454277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13704.450021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13709.369630] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13709.794905] LustreError: 454278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13714.914848] LustreError: 454277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13717.416789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13717.751592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13722.253865] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13729.929907] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13736.229713] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13756.478976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13756.825864] Lustre: Found index 0 for lustre-OST0000, updating log [13757.322988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13765.034812] LNet: 1 local NIs in recovery (showing 1): 192.168.206.107@tcp [13767.068630] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13777.323117] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13780.716383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13780.748191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13786.027812] Lustre: server umount lustre-OST0000 complete [13790.950236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13790.956624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13790.970068] Lustre: Skipped 1 previous similar message [13790.986494] Lustre: Skipped 1 previous similar message [13793.059018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13793.075197] Lustre: Skipped 3 previous similar messages [13796.227860] Lustre: server umount lustre-MDT0000 complete [13797.155762] LustreError: 454272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13797.189799] LustreError: 454272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13800.698880] LustreError: 454258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399476 with bad export cookie 12260959910731408005 [13800.705415] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13800.713825] LustreError: 454258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13801.242719] Lustre: server umount lustre-MDT0001 complete [13833.913115] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [13845.957703] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [13886.212876] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:52:40 (1787399560) [13905.228360] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [13908.243481] Key type lgssc unregistered [13908.590502] LNet: 460891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13908.606488] LNetError: 460891:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14018.215237] LNet: Removed LNI 192.168.206.107@tcp [14019.257361] Key type .llcrypt unregistered [14019.259148] Key type ._llcrypt unregistered [14037.016375] Key type ._llcrypt registered [14037.019024] Key type .llcrypt registered [14037.148221] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [14038.256261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14038.290056] alg: No test for adler32 (adler32-zlib) [14039.490479] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [14039.708514] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [14041.424436] Key type lgssc registered [14042.887366] Lustre: Echo OBD driver; http://www.lustre.org/ [14047.893919] LNet: Added LNI 192.168.206.107@tcp100 [8/256/0/180] [14049.002875] LNet: Added LNI 192.168.206.107@tcp101 [8/256/0/180] [14049.786995] LNet: Added LNI 192.168.206.107@tcp102 [8/256/0/180] [14050.725962] LNet: Added LNI 192.168.206.107@tcp103 [8/256/0/180] [14051.708221] LNet: Added LNI 192.168.206.107@tcp104 [8/256/0/180] [14052.883477] LNet: Added LNI 192.168.206.107@tcp105 [8/256/0/180] [14053.807666] LNet: Added LNI 192.168.206.107@tcp106 [8/256/0/180] [14055.008259] LNet: Added LNI 192.168.206.107@tcp107 [8/256/0/180] [14056.065350] LNet: Added LNI 192.168.206.107@tcp108 [8/256/0/180] [14056.965603] LNet: Added LNI 192.168.206.107@tcp109 [8/256/0/180] [14057.629540] LNet: Added LNI 192.168.206.107@tcp110 [8/256/0/180] [14058.396135] LNet: Added LNI 192.168.206.107@tcp111 [8/256/0/180] [14059.170189] LNet: Added LNI 192.168.206.107@tcp112 [8/256/0/180] [14060.406598] LNet: Added LNI 192.168.206.107@tcp113 [8/256/0/180] [14061.455549] LNet: Added LNI 192.168.206.107@tcp114 [8/256/0/180] [14062.503460] LNet: Added LNI 192.168.206.107@tcp115 [8/256/0/180] [14063.295625] LNet: Added LNI 192.168.206.107@tcp116 [8/256/0/180] [14064.089861] LNet: Added LNI 192.168.206.107@tcp117 [8/256/0/180] [14065.196991] LNet: Added LNI 192.168.206.107@tcp118 [8/256/0/180] [14065.995298] LNet: Added LNI 192.168.206.107@tcp119 [8/256/0/180] [14066.779262] LNet: Added LNI 192.168.206.107@tcp120 [8/256/0/180] [14067.881434] LNet: Added LNI 192.168.206.107@tcp121 [8/256/0/180] [14069.218913] LNet: Added LNI 192.168.206.107@tcp122 [8/256/0/180] [14070.349120] LNet: Added LNI 192.168.206.107@tcp123 [8/256/0/180] [14071.576598] LNet: Added LNI 192.168.206.107@tcp124 [8/256/0/180] [14072.814854] LNet: Added LNI 192.168.206.107@tcp125 [8/256/0/180] [14073.937986] LNet: Added LNI 192.168.206.107@tcp126 [8/256/0/180] [14075.038202] LNet: Added LNI 192.168.206.107@tcp127 [8/256/0/180] [14076.027322] LNet: Added LNI 192.168.206.107@tcp128 [8/256/0/180] [14076.943361] LNet: Added LNI 192.168.206.107@tcp129 [8/256/0/180] [14077.935767] LNet: Added LNI 192.168.206.107@tcp130 [8/256/0/180] [14079.017724] LNet: Added LNI 192.168.206.107@tcp131 [8/256/0/180] [14080.105478] LNet: Added LNI 192.168.206.107@tcp132 [8/256/0/180] [14081.144701] LNet: Added LNI 192.168.206.107@tcp133 [8/256/0/180] [14082.079603] LNet: Added LNI 192.168.206.107@tcp134 [8/256/0/180] [14083.229550] LNet: Added LNI 192.168.206.107@tcp135 [8/256/0/180] [14089.635795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14089.678111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14090.843651] Lustre: 463930:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.107@tcp: can't add NID 192.168.206.107@tcp130: rc = -75 [14090.931614] LNetError: 461390:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.107 [14090.944695] LNetError: Protocol error connecting to 192.168.206.107@tcp117 at host 192.168.206.107:988: is it running a compatible version of Lustre? [14090.956320] LNetError: Skipped 5 previous similar messages [14091.065315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14091.124585] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14091.251734] Lustre: lustre-MDT0000: new disk, initializing [14091.352568] Lustre: 463986:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.107@tcp: can't add NID 192.168.206.107@tcp130: rc = -75 [14091.363472] Lustre: 463986:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14091.384568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14091.409728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14091.937628] LNetError: 463954:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.107 [14091.940704] LNet: 9 local NIs in recovery (showing 9): 192.168.206.107@tcp123, 192.168.206.107@tcp124, 192.168.206.107@tcp121, 192.168.206.107@tcp125, 192.168.206.107@tcp117, 192.168.206.107@tcp127, 192.168.206.107@tcp126, 192.168.206.107@tcp128, 192.168.206.107@tcp129 [14091.947834] LNetError: 463954:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [14091.949404] LNetError: Protocol error connecting to 192.168.206.107@tcp104 at host 192.168.206.107:988: is it running a compatible version of Lustre? [14091.949412] LNetError: Skipped 7 previous similar messages [14093.091716] LNetError: 463952:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.107 [14093.105295] LNetError: 463952:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [14093.116222] LNetError: Protocol error connecting to 192.168.206.107@tcp127 at host 192.168.206.107:988: is it running a compatible version of Lustre? [14093.127769] LNetError: Skipped 10 previous similar messages [14096.347326] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14097.186316] LNetError: 461387:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.107 [14097.203860] LNetError: 461387:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14097.211874] LNetError: Protocol error connecting to 192.168.206.107@tcp127 at host 192.168.206.107:988: is it running a compatible version of Lustre? [14105.377585] LNetError: 463954:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.107 [14105.391839] LNetError: Protocol error connecting to 192.168.206.107@tcp127 at host 192.168.206.107:988: is it running a compatible version of Lustre? [14108.217803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14108.361361] Lustre: 463947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14108.432332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14108.438784] Lustre: Skipped 1 previous similar message [14108.571442] Lustre: lustre-MDT0001: new disk, initializing [14108.624815] Lustre: 464748:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.107@tcp: can't add NID 192.168.206.107@tcp130: rc = -75 [14108.631221] Lustre: 464748:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14108.884839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14108.914352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14108.929670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14113.955091] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14114.428995] Lustre: 465025:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.107@tcp: can't add NID 192.168.206.107@tcp130: rc = -75 [14114.441299] Lustre: 465025:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14120.614188] LNetError: 463951:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.107 [14120.629319] LNetError: Protocol error connecting to 192.168.206.107@tcp120 at host 192.168.206.107:988: is it running a compatible version of Lustre? [14125.150995] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14131.224942] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14136.996416] LNetError: 463952:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.107 [14137.015474] LNetError: 463952:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 14 previous similar messages [14137.029196] LNetError: Protocol error connecting to 192.168.206.107@tcp118 at host 192.168.206.107:988: is it running a compatible version of Lustre? [14137.044042] LNetError: Skipped 14 previous similar messages [14138.948847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14139.246987] Lustre: lustre-OST0000: new disk, initializing [14139.250258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14139.255705] Lustre: 465920:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14139.347326] Lustre: 465925:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.107@tcp: can't add NID 192.168.206.107@tcp130: rc = -75 [14139.358711] Lustre: 465925:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14139.411620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14145.171701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14145.198668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14145.366787] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14146.521484] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14151.456146] LNet: 1 local NIs in recovery (showing 1): 192.168.206.107@tcp127 [14158.199526] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14171.106575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14171.107962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14171.114160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14171.119075] Lustre: Skipped 1 previous similar message [14172.657783] Lustre: server umount lustre-OST0000 complete [14180.323226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14180.341679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14180.370141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14180.377922] Lustre: Skipped 1 previous similar message [14181.358288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14181.371747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14181.376709] Lustre: Skipped 1 previous similar message [14182.051479] LNetError: 463954:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.107 [14182.079435] LNetError: 463954:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 25 previous similar messages [14182.100158] LNetError: Protocol error connecting to 192.168.206.107@tcp104 at host 192.168.206.107:988: is it running a compatible version of Lustre? [14182.114439] LNetError: Skipped 25 previous similar messages [14183.095309] Lustre: server umount lustre-MDT0000 complete [14186.618032] LustreError: 463941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399862 with bad export cookie 4422342059273542227 [14186.626122] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14186.632159] LustreError: 463941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14187.226594] Lustre: server umount lustre-MDT0001 complete [14198.391986] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [14201.294625] Key type lgssc unregistered [14201.748775] LNet: 467833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14201.755515] LNetError: 467833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14201.760758] LNet: Removed LNI 192.168.206.107@tcp [14202.784746] LNet: Removed LNI 192.168.206.107@tcp100 [14202.787253] LNetError: 467833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14202.799937] LNetError: 467833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14202.811757] LNet: Removed LNI 192.168.206.107@tcp101 [14203.872466] LNet: Removed LNI 192.168.206.107@tcp102 [14203.875109] LNetError: 467833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14203.882514] LNetError: 467833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14203.889708] LNet: Removed LNI 192.168.206.107@tcp103 [14204.896721] LNet: Removed LNI 192.168.206.107@tcp104 [14205.921796] LNet: Removed LNI 192.168.206.107@tcp105 [14205.927644] LNetError: 467833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14205.942255] LNetError: 467833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14205.950926] LNet: Removed LNI 192.168.206.107@tcp106 [14207.009511] LNet: Removed LNI 192.168.206.107@tcp107 [14208.032274] LNet: Removed LNI 192.168.206.107@tcp108 [14209.056137] LNet: Removed LNI 192.168.206.107@tcp109 [14210.080363] LNet: Removed LNI 192.168.206.107@tcp110 [14210.087163] LNetError: 467833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14210.108881] LNetError: 467833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14210.116457] LNet: Removed LNI 192.168.206.107@tcp111 [14211.168270] LNet: Removed LNI 192.168.206.107@tcp112 [14212.193411] LNet: Removed LNI 192.168.206.107@tcp113 [14213.216246] LNet: Removed LNI 192.168.206.107@tcp114 [14214.240346] LNet: Removed LNI 192.168.206.107@tcp115 [14215.264204] LNet: Removed LNI 192.168.206.107@tcp116 [14216.291448] LNet: Removed LNI 192.168.206.107@tcp117 [14217.312150] LNet: Removed LNI 192.168.206.107@tcp118 [14218.336987] LNet: Removed LNI 192.168.206.107@tcp119 [14218.344714] LNetError: 467833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14218.352526] LNetError: 467833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14218.366421] LNet: Removed LNI 192.168.206.107@tcp120 [14219.424156] LNet: Removed LNI 192.168.206.107@tcp121 [14220.448156] LNet: Removed LNI 192.168.206.107@tcp122 [14221.472629] LNet: Removed LNI 192.168.206.107@tcp123 [14222.496174] LNet: Removed LNI 192.168.206.107@tcp124 [14223.520314] LNet: Removed LNI 192.168.206.107@tcp125 [14224.544733] LNet: Removed LNI 192.168.206.107@tcp126 [14224.549424] LNet: Removed LNI 192.168.206.107@tcp127 [14225.568506] LNet: Removed LNI 192.168.206.107@tcp128 [14226.592328] LNet: Removed LNI 192.168.206.107@tcp129 [14226.595565] LNet: Removed LNI 192.168.206.107@tcp130 [14226.611376] LNet: Removed LNI 192.168.206.107@tcp131 [14226.613345] LNet: Removed LNI 192.168.206.107@tcp132 [14226.628421] LNet: Removed LNI 192.168.206.107@tcp133 [14226.631371] LNet: Removed LNI 192.168.206.107@tcp134 [14226.649378] LNet: Removed LNI 192.168.206.107@tcp135 [14227.638344] Key type .llcrypt unregistered [14227.643161] Key type ._llcrypt unregistered [14256.902521] Key type ._llcrypt registered [14256.904808] Key type .llcrypt registered [14257.043032] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [14272.016901] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [14273.508228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14273.566357] alg: No test for adler32 (adler32-zlib) [14274.691742] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [14274.879584] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [14276.520319] Key type lgssc registered [14277.701965] Lustre: Echo OBD driver; http://www.lustre.org/ [14317.113291] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:59:51 (1787399991) [14334.950486] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [14337.775856] Key type lgssc unregistered [14338.040694] LNet: 471923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14338.048559] LNetError: 471923:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14338.069097] LNet: Removed LNI 192.168.206.107@tcp [14338.916469] Key type .llcrypt unregistered [14338.918202] Key type ._llcrypt unregistered [14354.643435] Key type ._llcrypt registered [14354.647790] Key type .llcrypt registered [14354.794583] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [14355.832343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14355.879187] alg: No test for adler32 (adler32-zlib) [14357.141312] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [14357.429255] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [14359.072400] Key type lgssc registered [14360.910841] Lustre: Echo OBD driver; http://www.lustre.org/ [14371.773803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14371.807821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14373.154382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14373.201430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14373.325551] Lustre: lustre-MDT0000: new disk, initializing [14373.467164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14373.498449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14378.450651] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14390.818642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14391.014162] Lustre: 473139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14391.084753] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14391.092091] Lustre: Skipped 1 previous similar message [14391.211346] Lustre: lustre-MDT0001: new disk, initializing [14391.327837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14391.394821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14391.417478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14396.296087] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14406.367980] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14412.466037] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14419.287460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14419.533418] Lustre: lustre-OST0000: new disk, initializing [14419.537807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14419.542818] Lustre: 475119:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14419.664686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14425.690224] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14426.194298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14426.211558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14426.300822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14434.964121] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14438.706388] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing lnet_if_list [14444.222786] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14444.254475] LNet: Added LNI 192.168.206.107@tcp10 [8/256/0/180] [14450.074551] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14450.103183] LNet: Added LNI 192.168.206.107@tcp11 [8/256/0/180] [14454.504831] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14454.533081] LNet: Added LNI 192.168.206.107@tcp12 [8/256/0/180] [14458.859491] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14458.900029] LNet: Added LNI 192.168.206.107@tcp13 [8/256/0/180] [14463.484503] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14463.511851] LNet: Added LNI 192.168.206.107@tcp14 [8/256/0/180] [14469.559878] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14469.604509] LNet: Added LNI 192.168.206.107@tcp15 [8/256/0/180] [14475.816293] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14475.864766] LNet: Added LNI 192.168.206.107@tcp16 [8/256/0/180] [14481.788889] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14481.824776] LNet: Added LNI 192.168.206.107@tcp17 [8/256/0/180] [14488.386761] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14488.429066] LNet: Added LNI 192.168.206.107@tcp18 [8/256/0/180] [14494.323270] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14494.381819] LNet: Added LNI 192.168.206.107@tcp19 [8/256/0/180] [14501.608714] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14501.664618] LNet: Added LNI 192.168.206.107@tcp20 [8/256/0/180] [14508.683067] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14508.739494] LNet: Added LNI 192.168.206.107@tcp21 [8/256/0/180] [14515.917238] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14516.016305] LNet: Added LNI 192.168.206.107@tcp22 [8/256/0/180] [14521.999621] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14522.025581] LNet: Added LNI 192.168.206.107@tcp23 [8/256/0/180] [14527.662321] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14527.693036] LNet: Added LNI 192.168.206.107@tcp24 [8/256/0/180] [14533.756694] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14533.795407] LNet: Added LNI 192.168.206.107@tcp25 [8/256/0/180] [14540.848477] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14540.879913] LNet: Added LNI 192.168.206.107@tcp26 [8/256/0/180] [14547.324870] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14547.356837] LNet: Added LNI 192.168.206.107@tcp27 [8/256/0/180] [14554.313941] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14554.343850] LNet: Added LNI 192.168.206.107@tcp28 [8/256/0/180] [14561.365767] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14561.422145] LNet: Added LNI 192.168.206.107@tcp29 [8/256/0/180] [14568.594362] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14568.624709] LNet: Added LNI 192.168.206.107@tcp30 [8/256/0/180] [14575.944986] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14575.989321] LNet: Added LNI 192.168.206.107@tcp31 [8/256/0/180] [14582.968565] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14583.037551] LNet: Added LNI 192.168.206.107@tcp32 [8/256/0/180] [14590.140631] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14590.178193] LNet: Added LNI 192.168.206.107@tcp33 [8/256/0/180] [14596.748866] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14596.770439] LNet: Added LNI 192.168.206.107@tcp34 [8/256/0/180] [14603.947089] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14603.991475] LNet: Added LNI 192.168.206.107@tcp35 [8/256/0/180] [14611.830721] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14611.862116] LNet: Added LNI 192.168.206.107@tcp36 [8/256/0/180] [14619.338756] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14619.384195] LNet: Added LNI 192.168.206.107@tcp37 [8/256/0/180] [14627.309479] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14627.329937] LNet: Added LNI 192.168.206.107@tcp38 [8/256/0/180] [14634.859522] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14634.897271] LNet: Added LNI 192.168.206.107@tcp39 [8/256/0/180] [14642.253293] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14642.296209] LNet: Added LNI 192.168.206.107@tcp40 [8/256/0/180] [14650.454927] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14650.481816] LNet: Added LNI 192.168.206.107@tcp41 [8/256/0/180] [14658.282959] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14658.329814] LNet: Added LNI 192.168.206.107@tcp42 [8/256/0/180] [14665.832382] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14665.879232] LNet: Added LNI 192.168.206.107@tcp43 [8/256/0/180] [14674.806811] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14674.843200] LNet: Added LNI 192.168.206.107@tcp44 [8/256/0/180] [14683.567658] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14683.595396] LNet: Added LNI 192.168.206.107@tcp45 [8/256/0/180] [14692.246153] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14692.270841] LNet: Added LNI 192.168.206.107@tcp46 [8/256/0/180] [14700.263273] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14700.291468] LNet: Added LNI 192.168.206.107@tcp47 [8/256/0/180] [14709.185383] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14709.224689] LNet: Added LNI 192.168.206.107@tcp48 [8/256/0/180] [14718.846518] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14718.914824] LNet: Added LNI 192.168.206.107@tcp49 [8/256/0/180] [14728.657824] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14728.721862] LNet: Added LNI 192.168.206.107@tcp50 [8/256/0/180] [14754.276169] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14754.281423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14754.294401] Lustre: Skipped 1 previous similar message [14754.327588] Lustre: Skipped 2 previous similar messages [14757.569557] Lustre: server umount lustre-MDT0000 complete [14762.734535] LustreError: 473129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400438 with bad export cookie 12370821508924072963 [14762.752968] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14762.761861] LustreError: 473129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14763.402896] Lustre: server umount lustre-MDT0001 complete [14780.898197] Lustre: 472411:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787400440/real 1787400440] req@0000000095d09beb x1874224777182848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787400456 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14780.928912] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14780.943032] Lustre: Skipped 1 previous similar message [14783.008166] Lustre: 472411:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787400443/real 1787400443] req@000000006bce0bbf x1874224777182976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787400459 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14784.214752] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [14784.992095] Lustre: 472409:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787400445/real 1787400445] req@00000000ddc2237d x1874224777183104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787400461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14788.064100] Lustre: 472408:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787400448/real 1787400448] req@00000000e8bc6140 x1874224777183360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787400464 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14792.903029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14793.186710] Lustre: 472410:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787400453/real 1787400453] req@0000000077852b79 x1874224777183488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787400469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14807.855884] LustreError: 475112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14807.869561] LustreError: 475112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14807.917708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14809.961693] LustreError: 475113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14809.964270] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14812.007826] LustreError: 475112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14812.290102] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14817.256266] LustreError: 475113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14821.845207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14822.284212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14826.431888] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14827.745239] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14836.088158] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14842.482389] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14863.335251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14863.348365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14863.368755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14864.357191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14864.376096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14867.426727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14867.435704] Lustre: Skipped 1 previous similar message [14872.552911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14872.574605] Lustre: Skipped 1 previous similar message [14877.152905] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14877.322225] Lustre: server umount lustre-OST0000 complete [14882.793622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14882.802884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14882.815057] Lustre: Skipped 2 previous similar messages [14882.830343] Lustre: Skipped 1 previous similar message [14887.582370] Lustre: server umount lustre-MDT0000 complete [14887.908295] LustreError: 499360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14887.934306] LustreError: 499360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14891.783707] LustreError: 473944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400567 with bad export cookie 12370821508924075581 [14891.795724] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14891.803787] LustreError: 473944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14892.333800] Lustre: server umount lustre-MDT0001 complete [14903.824104] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [14907.887383] Key type lgssc unregistered [14908.552229] LNet: 502165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14908.565647] LNetError: 502165:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14908.579604] LNet: Removed LNI 192.168.206.107@tcp [14908.585724] LNet: Removed LNI 192.168.206.107@tcp10 [14908.593279] LNet: Removed LNI 192.168.206.107@tcp11 [14908.597930] LNet: Removed LNI 192.168.206.107@tcp12 [14908.602405] LNet: Removed LNI 192.168.206.107@tcp13 [14908.607335] LNet: Removed LNI 192.168.206.107@tcp14 [14908.618849] LNet: Removed LNI 192.168.206.107@tcp15 [14908.630872] LNet: Removed LNI 192.168.206.107@tcp16 [14908.637259] LNet: Removed LNI 192.168.206.107@tcp17 [14908.640341] LNet: Removed LNI 192.168.206.107@tcp18 [14908.670929] LNet: Removed LNI 192.168.206.107@tcp19 [14908.681502] LNet: Removed LNI 192.168.206.107@tcp20 [14908.700241] LNet: Removed LNI 192.168.206.107@tcp21 [14908.702517] LNet: Removed LNI 192.168.206.107@tcp22 [14908.715678] LNet: Removed LNI 192.168.206.107@tcp23 [14908.736753] LNet: Removed LNI 192.168.206.107@tcp24 [14908.738712] LNet: Removed LNI 192.168.206.107@tcp25 [14908.751451] LNet: Removed LNI 192.168.206.107@tcp26 [14908.753426] LNet: Removed LNI 192.168.206.107@tcp27 [14908.771143] LNet: Removed LNI 192.168.206.107@tcp28 [14908.773265] LNet: Removed LNI 192.168.206.107@tcp29 [14908.804864] LNet: Removed LNI 192.168.206.107@tcp30 [14908.814666] LNet: Removed LNI 192.168.206.107@tcp31 [14908.823401] LNet: Removed LNI 192.168.206.107@tcp32 [14908.843421] LNet: Removed LNI 192.168.206.107@tcp33 [14908.851965] LNet: Removed LNI 192.168.206.107@tcp34 [14908.858633] LNet: Removed LNI 192.168.206.107@tcp35 [14908.870382] LNet: Removed LNI 192.168.206.107@tcp36 [14908.872318] LNet: Removed LNI 192.168.206.107@tcp37 [14908.888407] LNet: Removed LNI 192.168.206.107@tcp38 [14908.894350] LNet: Removed LNI 192.168.206.107@tcp39 [14908.895836] LNet: Removed LNI 192.168.206.107@tcp40 [14908.900591] LNet: Removed LNI 192.168.206.107@tcp41 [14908.909804] LNet: Removed LNI 192.168.206.107@tcp42 [14908.913157] LNet: Removed LNI 192.168.206.107@tcp43 [14908.918801] LNet: Removed LNI 192.168.206.107@tcp44 [14908.923901] LNet: Removed LNI 192.168.206.107@tcp45 [14908.933095] LNet: Removed LNI 192.168.206.107@tcp46 [14908.942559] LNet: Removed LNI 192.168.206.107@tcp47 [14908.954449] LNet: Removed LNI 192.168.206.107@tcp48 [14908.961212] LNet: Removed LNI 192.168.206.107@tcp49 [14908.992919] LNet: Removed LNI 192.168.206.107@tcp50 [14909.812366] Key type .llcrypt unregistered [14909.815217] Key type ._llcrypt unregistered [14936.845816] Key type ._llcrypt registered [14936.851554] Key type .llcrypt registered [14937.048938] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [14954.323341] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [14955.591141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14955.673182] alg: No test for adler32 (adler32-zlib) [14956.852320] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [14957.113276] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [14958.784959] Key type lgssc registered [14959.968815] Lustre: Echo OBD driver; http://www.lustre.org/ [14997.760237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14997.788328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14999.061782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14999.098200] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14999.174098] Lustre: lustre-MDT0000: new disk, initializing [14999.274919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14999.295544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15004.502155] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15017.228024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15017.370693] Lustre: 505365:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15017.474988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15017.480986] Lustre: Skipped 1 previous similar message [15017.630237] Lustre: lustre-MDT0001: new disk, initializing [15017.740212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15017.785933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15017.793853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15023.448456] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15032.583640] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15038.451884] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15047.310440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15047.591323] Lustre: *** cfs_fail_loc=90f, val=0*** [15047.661609] Lustre: lustre-OST0000: new disk, initializing [15047.664767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15047.684751] Lustre: 507384:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15047.787202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15056.156682] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15056.992277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15057.007360] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15057.155350] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15067.039371] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15077.857035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15077.863163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15077.870991] LustreError: Skipped 1 previous similar message [15077.873937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15077.900374] Lustre: Skipped 1 previous similar message [15082.977138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15082.990421] Lustre: Skipped 1 previous similar message [15088.104655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15088.118981] Lustre: Skipped 2 previous similar messages [15089.120499] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15089.327992] Lustre: server umount lustre-OST0000 complete [15094.759080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15094.771122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15094.799922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15098.339823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15099.808795] Lustre: server umount lustre-MDT0000 complete [15104.008192] LustreError: 507383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400780 with bad export cookie 7297205298150599450 [15104.011700] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15104.028689] LustreError: 507383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15104.583397] Lustre: server umount lustre-MDT0001 complete [15116.402972] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [15119.425671] Key type lgssc unregistered [15119.670764] LNet: 509290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15119.686354] LNetError: 509290:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15119.704204] LNet: Removed LNI 192.168.206.107@tcp [15120.470521] Key type .llcrypt unregistered [15120.473734] Key type ._llcrypt unregistered [15140.644383] Key type ._llcrypt registered [15140.649830] Key type .llcrypt registered [15140.831946] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [15142.463525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15142.493829] alg: No test for adler32 (adler32-zlib) [15143.531346] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [15143.768673] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [15145.472182] Key type lgssc registered [15146.693698] Lustre: Echo OBD driver; http://www.lustre.org/ [15155.708199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15155.728446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15157.184425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15161.661959] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15167.458231] LustreError: 510562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15167.488143] LustreError: 510562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15170.174366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15170.540478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15175.981711] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15183.916606] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15189.993377] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15197.784912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15198.250352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15205.692888] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15214.111388] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15223.806995] loop0: detected capacity change from 0 to 204800000 [15224.031854] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15224.042358] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15224.059880] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15230.410061] loop0: detected capacity change from 0 to 204800000 [15230.643933] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15230.657194] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15230.675838] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15236.577358] loop0: detected capacity change from 0 to 204800000 [15242.650871] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15242.870698] Lustre: fs15246-OST0000: new disk, initializing [15242.877853] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15242.903705] Lustre: 514157:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15243.000688] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15249.931565] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15260.398594] loop1: detected capacity change from 0 to 204800000 [15266.284111] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15266.409547] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15266.476272] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15266.588846] Lustre: fs15246-MDT0000: new disk, initializing [15266.819710] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15266.869928] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15267.060889] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15268.129438] Lustre: 515281:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000007e449534 x1874225601231872/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787400960 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15271.697471] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15277.290630] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15277.305098] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15277.370382] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15301.107101] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15301.123300] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15303.019320] Lustre: server umount fs15246-MDT0000 complete [15306.217578] LustreError: 514589:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15311.338533] LustreError: 512124:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15312.586989] Lustre: server umount fs15246-OST0000 complete [15325.154795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15325.162982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15325.183210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15326.693746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15326.700545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15330.828827] Lustre: server umount lustre-OST0000 complete [15336.931578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15336.944697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15336.946538] Lustre: Skipped 1 previous similar message [15341.490395] Lustre: server umount lustre-MDT0000 complete [15342.051453] LustreError: 510557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15342.080880] LustreError: 510557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15345.863756] LustreError: 510542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401021 with bad export cookie 12792157947963752927 [15345.877728] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15345.886125] LustreError: 510542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15346.741495] Lustre: server umount lustre-MDT0001 complete [15362.411094] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [15365.719660] Key type lgssc unregistered [15366.051196] LNet: 518379:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15366.059973] LNetError: 518379:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15366.075953] LNet: Removed LNI 192.168.206.107@tcp [15366.900934] Key type .llcrypt unregistered [15366.904172] Key type ._llcrypt unregistered [15386.523430] Key type ._llcrypt registered [15386.534545] Key type .llcrypt registered [15386.654478] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [15387.765327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15387.822206] alg: No test for adler32 (adler32-zlib) [15388.855581] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [15389.096451] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [15390.745180] Key type lgssc registered [15392.127786] Lustre: Echo OBD driver; http://www.lustre.org/ [15433.650161] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [15444.750618] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [15482.155935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15482.202946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15483.473472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15483.515089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15483.596552] Lustre: lustre-MDT0000: new disk, initializing [15483.661127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15483.677868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15488.198636] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15499.381548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15499.488759] Lustre: 522847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15499.550184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15499.555340] Lustre: Skipped 1 previous similar message [15499.647889] Lustre: lustre-MDT0001: new disk, initializing [15499.739115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15499.791286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15499.804672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15504.057137] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15513.576672] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15518.993800] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15525.762845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15526.041335] Lustre: lustre-OST0000: new disk, initializing [15526.049511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15526.062571] Lustre: 524817:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15526.156670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15532.240139] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15532.674646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15532.690256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15532.774135] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15542.123979] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15546.348718] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15546.520530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15550.119885] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15550.257521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15553.519317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15553.525040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15553.531032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15553.550677] Lustre: Skipped 1 previous similar message [15558.635301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15558.647586] Lustre: Skipped 2 previous similar messages [15563.751687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15563.771967] Lustre: Skipped 1 previous similar message [15566.308110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15566.532217] Lustre: server umount lustre-OST0000 complete [15571.425128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15571.432830] LustreError: Skipped 1 previous similar message [15571.438428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15571.449824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15574.002036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15577.130972] Lustre: server umount lustre-MDT0000 complete [15580.964173] LustreError: 522840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401257 with bad export cookie 8433063893444859984 [15580.973716] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15580.977249] LustreError: 522840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15581.405963] Lustre: server umount lustre-MDT0001 complete [15609.155738] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [15620.084065] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [15647.557178] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15347 sec ========= 08:22:02 (1787401322) [15649.016290] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:22:03 (1787401323) === [15652.081389] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:22:06 (1787401326) === [15687.605686] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [15690.229360] Key type lgssc unregistered [15690.518577] LNet: 531460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15690.528511] LNetError: 531460:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15690.554969] LNet: Removed LNI 192.168.206.107@tcp [15691.624476] Key type .llcrypt unregistered [15691.626731] Key type ._llcrypt unregistered