[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 564804578 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001019] APIC: Switch to symmetric I/O mode setup [ 0.002342] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.007782] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011134] LSM: Security Framework initializing [ 0.012074] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014083] *** VALIDATE selinux *** [ 0.023384] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028515] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031103] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032138] *** VALIDATE tmpfs *** [ 0.034512] *** VALIDATE proc *** [ 0.036141] *** VALIDATE cgroup *** [ 0.037010] *** VALIDATE cgroup2 *** [ 0.038279] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041037] Spectre V2 : User space: Vulnerable [ 0.042011] Speculative Store Bypass: Vulnerable [ 0.045268] debug: unmapping init [mem 0xffffffffb5c59000-0xffffffffb5c60fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048789] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050016] ... bit width: 48 [ 0.051014] ... generic registers: 4 [ 0.052021] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054014] ... fixed-purpose events: 3 [ 0.055015] ... event mask: 000000070000000f [ 0.056316] rcu: Hierarchical SRCU implementation. [ 0.058561] smp: Bringing up secondary CPUs ... [ 0.059644] x86: Booting SMP configuration: [ 0.060034] .... node #0, CPUs: #1 #2 #3 [ 0.063508] smp: Brought up 1 node, 4 CPUs [ 0.065011] smpboot: Max logical packages: 1 [ 0.066018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.215896] node 0 deferred pages initialised in 147ms [ 0.221138] devtmpfs: initialized [ 0.222356] x86/mm: Memory block size: 128MB [ 0.226340] gcov: version magic: 0x41383552 [ 0.229334] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.230099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.231418] pinctrl core: initialized pinctrl subsystem [ 0.232377] [ 0.233000] ************************************************************* [ 0.235023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.238015] ** ** [ 0.241013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.244091] ** ** [ 0.246022] ** This means that this kernel is built to expose internal ** [ 0.248016] ** IOMMU data structures, which may compromise security on ** [ 0.251022] ** your system. ** [ 0.254015] ** ** [ 0.257015] ** If you see this message and you are not debugging the ** [ 0.260021] ** kernel, report this immediately to your vendor! ** [ 0.262025] ** ** [ 0.264022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.266013] ************************************************************* [ 0.270017] NET: Registered protocol family 16 [ 0.272471] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.277146] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.281146] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.286161] cpuidle: using governor menu [ 0.287660] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.288472] PCI: Using configuration type 1 for base access [ 0.289120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.298136] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.301082] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.306104] cryptd: max_cpu_qlen set to 1000 [ 0.310322] ACPI: Added _OSI(Module Device) [ 0.312044] ACPI: Added _OSI(Processor Device) [ 0.315043] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.318022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.325605] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.338796] ACPI: Interpreter enabled [ 0.342095] ACPI: PM: (supports S0 S3 S4 S5) [ 0.344033] ACPI: Using IOAPIC for interrupt routing [ 0.345232] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.349543] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.361436] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.364072] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.367034] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.371125] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.378128] acpiphp: Slot [2] registered [ 0.379114] acpiphp: Slot [5] registered [ 0.381157] acpiphp: Slot [6] registered [ 0.382105] acpiphp: Slot [7] registered [ 0.383072] acpiphp: Slot [8] registered [ 0.384125] acpiphp: Slot [9] registered [ 0.386119] acpiphp: Slot [10] registered [ 0.387145] acpiphp: Slot [3] registered [ 0.389148] acpiphp: Slot [4] registered [ 0.390132] acpiphp: Slot [11] registered [ 0.392144] acpiphp: Slot [12] registered [ 0.393119] acpiphp: Slot [13] registered [ 0.395136] acpiphp: Slot [14] registered [ 0.397189] acpiphp: Slot [15] registered [ 0.398155] acpiphp: Slot [16] registered [ 0.400169] acpiphp: Slot [17] registered [ 0.401200] acpiphp: Slot [18] registered [ 0.403134] acpiphp: Slot [19] registered [ 0.404159] acpiphp: Slot [20] registered [ 0.406161] acpiphp: Slot [21] registered [ 0.407144] acpiphp: Slot [22] registered [ 0.408130] acpiphp: Slot [23] registered [ 0.409097] acpiphp: Slot [24] registered [ 0.410119] acpiphp: Slot [25] registered [ 0.412180] acpiphp: Slot [26] registered [ 0.413165] acpiphp: Slot [27] registered [ 0.415149] acpiphp: Slot [28] registered [ 0.416060] acpiphp: Slot [29] registered [ 0.417044] acpiphp: Slot [30] registered [ 0.417982] acpiphp: Slot [31] registered [ 0.419078] PCI host bridge to bus 0000:00 [ 0.420012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.421039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.422031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.425035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.427050] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.428035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.429094] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.432116] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.435608] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.445812] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.451648] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.454044] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.456023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.457032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.461245] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.465869] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.468078] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.472034] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.478033] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.491048] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.497027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.506920] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.519062] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.530034] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.557031] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.571179] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.583042] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.597059] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.616044] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.629641] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.644022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.659038] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.700044] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.719147] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.728032] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.740044] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.763034] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.773332] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.780049] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.786026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.806044] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.817000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.833026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.846038] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.862035] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.872709] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.875521] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.878477] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.881578] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.883224] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.889063] iommu: Default domain type: Passthrough [ 0.890548] SCSI subsystem initialized [ 0.891225] ACPI: bus type USB registered [ 0.893182] usbcore: registered new interface driver usbfs [ 0.895153] usbcore: registered new interface driver hub [ 0.897113] usbcore: registered new device driver usb [ 0.898265] pps_core: LinuxPPS API ver. 1 registered [ 0.900019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.904118] PTP clock support registered [ 0.906063] EDAC MC: Ver: 3.0.0 [ 0.907670] PCI: Using ACPI for IRQ routing [ 0.909708] NetLabel: Initializing [ 0.911021] NetLabel: domain hash size = 128 [ 0.912013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.914153] NetLabel: unlabeled traffic allowed by default [ 0.916229] vgaarb: loaded [ 0.917417] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.919031] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.928000] clocksource: Switched to clocksource kvm-clock [ 1.074141] VFS: Disk quotas dquot_6.6.0 [ 1.077275] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.081704] *** VALIDATE ramfs *** [ 1.082690] *** VALIDATE hugetlbfs *** [ 1.083935] pnp: PnP ACPI init [ 1.086122] pnp: PnP ACPI: found 6 devices [ 1.101273] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.103894] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.105620] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.107342] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.109225] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.111094] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.113399] NET: Registered protocol family 2 [ 1.115503] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.119605] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.122499] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.127359] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.130304] TCP: Hash tables configured (established 65536 bind 65536) [ 1.132777] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.135282] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.137693] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.140315] NET: Registered protocol family 1 [ 1.142391] RPC: Registered named UNIX socket transport module. [ 1.144732] RPC: Registered udp transport module. [ 1.146600] RPC: Registered tcp transport module. [ 1.147724] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.149914] NET: Registered protocol family 44 [ 1.151200] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.153400] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.155734] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.158412] PCI: CLS 0 bytes, default 64 [ 1.161325] Unpacking initramfs... [ 2.557309] debug: unmapping init [mem 0xffff9e433cc54000-0xffff9e433ffbffff] [ 2.561476] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.564172] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.567287] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.062590] Initialise system trusted keyrings [ 3.064274] Key type blacklist registered [ 3.066772] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.077117] zbud: loaded [ 3.080789] *** VALIDATE nfs *** [ 3.082694] *** VALIDATE nfs4 *** [ 3.084427] pstore: using deflate compression [ 3.088180] Platform Keyring initialized [ 3.193226] NET: Registered protocol family 38 [ 3.195331] Key type asymmetric registered [ 3.197081] Asymmetric key parser 'x509' registered [ 3.199124] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.202511] io scheduler mq-deadline registered [ 3.204310] io scheduler kyber registered [ 3.205731] io scheduler bfq registered [ 3.207664] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.210259] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.213167] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.215742] ACPI: Power Button [PWRF] [ 3.220948] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.227726] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.240128] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.249321] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.262739] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.292806] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.320691] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.325682] Non-volatile memory driver v1.3 [ 3.327221] Linux agpgart interface v0.103 [ 3.363859] virtio_blk virtio1: [vda] 149800 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.366864] vda: detected capacity change from 0 to 76697600 [ 3.397237] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.400363] vdb: detected capacity change from 0 to 1073741824 [ 3.414355] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.416672] vdc: detected capacity change from 0 to 2621440000 [ 3.430640] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.433474] vdd: detected capacity change from 0 to 2621440000 [ 3.448224] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.451332] vde: detected capacity change from 0 to 4294967296 [ 3.465795] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.468716] vdf: detected capacity change from 0 to 4294967296 [ 3.474725] libphy: Fixed MDIO Bus: probed [ 3.486842] usbcore: registered new interface driver usbserial_generic [ 3.489696] usbserial: USB Serial support registered for generic [ 3.491565] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.496254] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.498431] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.501359] mousedev: PS/2 mouse device common for all mice [ 3.504308] rtc_cmos 00:05: RTC can wake from S4 [ 3.507714] rtc_cmos 00:05: registered as rtc0 [ 3.510457] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.510728] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.512328] intel_pstate: CPU model not supported [ 3.516931] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.523821] hid: raw HID events driver (C) Jiri Kosina [ 3.525377] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.526216] usbcore: registered new interface driver usbhid [ 3.529499] usbhid: USB HID core driver [ 3.530818] drop_monitor: Initializing network drop monitor service [ 3.532726] Initializing XFRM netlink socket [ 3.534689] NET: Registered protocol family 10 [ 3.538995] Segment Routing with IPv6 [ 3.540542] NET: Registered protocol family 17 [ 3.542507] mpls_gso: MPLS GSO support [ 3.546920] RAS: Correctable Errors collector initialized. [ 3.548514] AVX version of gcm_enc/dec engaged. [ 3.549550] AES CTR mode by8 optimization enabled [ 3.645943] sched_clock: Marking stable (3645902318, 0)->(4700079720, -1054177402) [ 3.649060] registered taskstats version 1 [ 3.651302] Loading compiled-in X.509 certificates [ 3.653606] zswap: loaded using pool lzo/zbud [ 3.688862] Key type big_key registered [ 3.701318] Key type encrypted registered [ 3.702750] ima: No TPM chip found, activating TPM-bypass! [ 3.704868] ima: Allocated hash algorithm: sha1 [ 3.706894] ima: No architecture policies found [ 3.708904] evm: Initialising EVM extended attributes: [ 3.710869] evm: security.selinux [ 3.712231] evm: security.ima [ 3.713102] evm: security.capability [ 3.714280] evm: HMAC attrs: 0x1 [ 3.716622] rtc_cmos 00:05: setting system clock to 2026-09-04 13:49:46 UTC (1788529786) [ 3.723599] debug: unmapping init [mem 0xffffffffb6c03000-0xffffffffb6dfffff] [ 3.726907] debug: unmapping init [mem 0xffffffffb5982000-0xffffffffb5c58fff] [ 3.735150] Write protecting the kernel read-only data: 28672k [ 3.739269] debug: unmapping init [mem 0xffffffffb4003000-0xffffffffb41fffff] [ 3.742250] debug: unmapping init [mem 0xffffffffb4914000-0xffffffffb49fffff] [ 3.781639] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.789876] systemd[1]: Detected virtualization kvm. [ 3.791891] systemd[1]: Detected architecture x86-64. [ 3.793893] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.821858] systemd[1]: No hostname configured. [ 3.823490] systemd[1]: Set hostname to . [ 3.825521] random: systemd: uninitialized urandom read (16 bytes read) [ 3.827899] systemd[1]: Initializing machine ID from random generator. [ 3.977935] random: systemd: uninitialized urandom read (16 bytes read) [ 3.980737] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.984881] random: systemd: uninitialized urandom read (16 bytes read) [ 3.987957] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.997666] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.680571] device-mapper: uevent: version 1.0.3 [ 4.682881] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 5.073941] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.429642] virtio_net virtio0 ens2: renamed from eth0 [ 5.463408] scsi host0: ata_piix [ 5.516267] scsi host1: ata_piix [ 5.518852] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.521727] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.481821] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.219882] random: crng init done [ 10.221442] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.900188] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.215894] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.481861] SELinux: Disabled at runtime. [ 12.541949] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.549890] systemd[1]: Detected virtualization kvm. [ 12.551520] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.106737] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.110151] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.115717] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.119763] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.124213] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.135967] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.142634] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward [ 13.230200] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.785941] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.127326] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.176101] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.304448] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.327217] EDAC sbridge: Ver: 1.1.2 [ 16.006288] Key type dns_resolver registered [ 16.315173] NFS: Registering the id_resolver key type [ 16.316990] Key type id_resolver registered [ 16.318822] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg226-server login: [ 29.440121] hrtimer: interrupt took 2566629 ns [ 87.773922] libcfs: loading out-of-tree module taints kernel. [ 87.844346] Key type ._llcrypt registered [ 87.847043] Key type .llcrypt registered [ 87.980041] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 109.678397] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 112.077753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 112.117608] alg: No test for adler32 (adler32-zlib) [ 113.963556] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 115.086859] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 116.944894] Key type lgssc registered [ 119.499310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.966968] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 200.048918] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 217.146169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 217.210640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 218.627352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 218.711701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 218.801724] Lustre: lustre-MDT0000: new disk, initializing [ 218.886679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.914442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 224.871756] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.495940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 242.617809] Lustre: 6514:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 242.759882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 242.767644] Lustre: Skipped 1 previous similar message [ 242.927638] Lustre: lustre-MDT0001: new disk, initializing [ 243.021900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.058937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 243.072613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 248.701689] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.447746] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 266.716993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.332233] Lustre: lustre-OST0000: new disk, initializing [ 267.337845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 267.350609] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 267.447059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 274.484569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 274.493772] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 274.609443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 274.759991] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 291.609511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 291.866868] Lustre: lustre-OST0001: new disk, initializing [ 291.873216] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 291.886711] Lustre: 9528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 291.996456] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 299.374260] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 299.682183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 299.696659] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 299.811263] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 313.582950] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 323.791546] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 332.324734] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing check_logdir /tmp/testlogs/ [ 339.000064] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing yml_node [ 344.252373] Lustre: DEBUG MARKER: Client: 2.17.58.5 [ 347.481938] Lustre: DEBUG MARKER: MDS: 2.17.58.5 [ 350.785586] Lustre: DEBUG MARKER: OSS: 2.17.58.5 [ 353.035956] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 09:55:33 EDT 2026 [ 374.108081] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 376.158770] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 396.777925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 396.791277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 396.819634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 402.449105] Lustre: server umount lustre-MDT0000 complete [ 407.074831] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530189 with bad export cookie 11252716931630603988 [ 407.077333] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 407.104090] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 407.538082] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 407.540621] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 407.554890] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 407.554900] Lustre: Skipped 4 previous similar messages [ 407.561957] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 407.579885] Lustre: Skipped 4 previous similar messages [ 412.643278] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.647526] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 412.693860] Lustre: Skipped 2 previous similar messages [ 413.667152] Lustre: server umount lustre-MDT0001 complete [ 417.820697] Lustre: server umount lustre-OST0000 complete [ 422.750560] Lustre: server umount lustre-OST0001 complete [ 434.534909] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 447.465082] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 490.888897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.293144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 491.362731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 491.580816] Lustre: lustre-MDT0000: new disk, initializing [ 491.781570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.813696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 498.820647] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 512.007982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.102291] Lustre: 15085:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 512.214830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 512.220457] Lustre: Skipped 1 previous similar message [ 512.349539] Lustre: lustre-MDT0001: new disk, initializing [ 512.479946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 512.558547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 512.588354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 517.835405] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 528.307497] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 536.125433] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 545.183285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 545.752286] Lustre: lustre-OST0000: new disk, initializing [ 545.759753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 545.779196] Lustre: 17053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 546.011886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.460143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 547.482706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 547.679480] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 554.545492] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 565.506851] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 571.658846] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 571.955294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 577.074913] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 577.327864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 583.649655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.653676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 583.664899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 585.845470] Lustre: server umount lustre-OST0000 complete [ 593.892247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.914488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 593.916207] Lustre: Skipped 2 previous similar messages [ 593.937466] Lustre: Skipped 2 previous similar messages [ 596.549811] Lustre: server umount lustre-MDT0000 complete [ 599.009442] LustreError: 15092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.019749] LustreError: 15092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 601.348608] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530384 with bad export cookie 11252716931630605668 [ 601.371607] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 601.378589] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 601.872385] Lustre: server umount lustre-MDT0001 complete [ 607.907320] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:59:48 (1788530388) [ 610.869477] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 612.701160] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:59:54 (1788530394) [ 621.288969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.920191] LustreError: 19465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 622.032222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 627.171841] LustreError: 19466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.137285] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 636.391940] LustreError: 19465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 636.416923] LustreError: 19465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 640.356238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.039123] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 656.682948] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 663.868714] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 673.851580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 683.367831] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 694.799853] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 707.049544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 707.051490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.056084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 707.060409] LustreError: Skipped 1 previous similar message [ 707.092047] Lustre: Skipped 1 previous similar message [ 712.061942] Lustre: server umount lustre-OST0000 complete [ 717.299607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 717.320602] Lustre: Skipped 1 previous similar message [ 726.498418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 726.507399] Lustre: Skipped 6 previous similar messages [ 731.104162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 731.358597] Lustre: server umount lustre-MDT0000 complete [ 731.632375] LustreError: 19465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 736.196294] LustreError: 19447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530518 with bad export cookie 11252716931630606599 [ 736.200128] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 736.218047] LustreError: 19447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 736.674238] Lustre: server umount lustre-MDT0001 complete [ 752.982533] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 756.758788] Key type lgssc unregistered [ 757.050531] LNet: 23019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.056964] LNetError: 23019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.072797] LNet: Removed LNI 192.168.202.126@tcp [ 758.430865] Key type .llcrypt unregistered [ 758.435220] Key type ._llcrypt unregistered [ 784.162509] Key type ._llcrypt registered [ 784.164419] Key type .llcrypt registered [ 784.377874] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 785.382165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 785.437613] alg: No test for adler32 (adler32-zlib) [ 787.182762] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 787.601299] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 789.464168] Key type lgssc registered [ 791.205075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 803.210142] loop: module loaded [ 803.341126] loop0: detected capacity change from 0 to 204800000 [ 803.751041] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 803.759545] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 804.166573] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 811.823125] loop0: detected capacity change from 0 to 204800000 [ 812.131173] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 812.148332] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 812.171765] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 817.290210] loop0: detected capacity change from 0 to 204800000 [ 823.766629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 823.807474] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.207695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 825.310904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 825.394467] Lustre: lustre-MDT0000: new disk, initializing [ 825.501078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 825.528801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 830.676100] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 840.414531] loop1: detected capacity change from 0 to 204800000 [ 846.125775] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 846.463747] Lustre: lustre-OST07c6: new disk, initializing [ 846.468954] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 846.473470] Lustre: Skipped 1 previous similar message [ 846.486774] Lustre: 26203:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 846.563387] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 853.873956] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 854.632255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 854.673383] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 854.704493] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 859.764220] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 859.780729] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 859.794110] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 859.810429] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 859.826593] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 859.837053] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 880.083092] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 880.091806] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 880.102286] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 880.118396] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 880.126597] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 880.140868] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 885.136500] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 885.150178] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 885.169434] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 885.174304] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 885.189853] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 885.201428] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 885.214803] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 885.227949] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 885.233682] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 885.240346] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 885.245569] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 885.253357] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 888.555472] Lustre: 26820:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 888.578616] Lustre: 26820:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 888.590311] Lustre: 26820:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 888.602700] Lustre: 26820:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 888.613307] Lustre: 26820:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 888.621924] Lustre: 26820:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 888.629490] Lustre: 26820:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 888.642397] Lustre: 26820:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 888.648524] Lustre: 26820:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 888.665497] Lustre: 26820:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 888.671977] Lustre: 26820:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 888.680280] Lustre: 26820:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 893.492080] Lustre: 26820:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 893.496698] Lustre: 26820:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 893.509676] Lustre: 26820:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 893.523028] Lustre: 26820:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 893.532015] Lustre: 26820:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 893.540030] Lustre: 26820:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 893.546520] Lustre: 26820:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 893.554593] Lustre: 26820:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 893.561659] Lustre: 26820:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 893.568267] Lustre: 26820:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 893.574463] Lustre: 26820:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 893.580924] Lustre: 26820:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 901.777878] Lustre: 25190:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 901.788339] Lustre: 25190:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 901.795149] Lustre: 25190:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 901.800457] Lustre: 25190:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 901.807217] Lustre: 25190:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 901.815856] Lustre: 25190:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 901.822469] Lustre: 25190:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 901.829274] Lustre: 25190:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 901.839698] Lustre: 25190:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 901.850254] Lustre: 25190:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 901.865413] Lustre: 25190:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 901.880449] Lustre: 25190:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 918.068969] Lustre: 25190:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 918.077065] Lustre: 25190:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 918.084459] Lustre: 25190:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 918.089397] Lustre: 25190:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 918.103105] Lustre: 25190:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 918.110702] Lustre: 25190:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 918.124366] Lustre: 25190:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 918.135314] Lustre: 25190:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 918.143445] Lustre: 25190:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 918.149078] Lustre: 25190:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 918.157434] Lustre: 25190:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 918.163564] Lustre: 25190:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 956.899929] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 956.917926] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.958659] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 960.570396] Lustre: server umount lustre-OST07c6 complete [ 971.466542] Lustre: server umount lustre-MDT0000 complete [ 989.801680] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 993.557690] Key type lgssc unregistered [ 993.923530] LNet: 28766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.929600] LNetError: 28766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.940436] LNet: Removed LNI 192.168.202.126@tcp [ 995.068617] Key type .llcrypt unregistered [ 995.074397] Key type ._llcrypt unregistered [ 1014.978834] Key type ._llcrypt registered [ 1014.982586] Key type .llcrypt registered [ 1015.126821] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 1016.453563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1016.567629] alg: No test for adler32 (adler32-zlib) [ 1017.629278] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 1017.916729] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 1019.641340] Key type lgssc registered [ 1020.761641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1029.363276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1029.377204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1030.876756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.735407] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1040.358750] LustreError: 30036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.379321] LustreError: 30036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1045.482425] LustreError: 30037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.768827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1046.101432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1051.094754] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1059.857880] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1067.638775] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1075.587889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1076.005204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1077.056825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1081.785897] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1090.328333] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1107.937487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1107.944815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.966306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1109.564517] Lustre: server umount lustre-OST0000 complete [ 1117.154570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.156725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1117.162907] Lustre: Skipped 1 previous similar message [ 1120.201491] Lustre: server umount lustre-MDT0000 complete [ 1122.281162] LustreError: 31020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1122.310588] LustreError: 31020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1124.119365] LustreError: 30019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530906 with bad export cookie 4385666283321451096 [ 1124.133903] LustreError: 30019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1124.136018] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1134.027503] Lustre: server umount lustre-MDT0001 complete [ 1145.797210] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 1149.306436] Key type lgssc unregistered [ 1149.781410] LNet: 33772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.796391] LNetError: 33772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1149.850311] LNet: Removed LNI 192.168.202.126@tcp [ 1150.812993] Key type .llcrypt unregistered [ 1150.825439] Key type ._llcrypt unregistered [ 1174.029910] Key type ._llcrypt registered [ 1174.032815] Key type .llcrypt registered [ 1174.155771] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 1175.578658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1175.670127] alg: No test for adler32 (adler32-zlib) [ 1176.906819] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 1177.227278] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 1178.896545] Key type lgssc registered [ 1180.246632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1188.707277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1188.735176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.431196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.880566] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1200.611118] LustreError: 35039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1200.641656] LustreError: 35039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1201.634829] LustreError: 35040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.617044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1206.120890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1211.151514] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1220.064807] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1227.956555] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1235.092049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1235.654096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.670803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1242.903790] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1250.853707] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1261.544057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1261.555632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.559262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1261.574184] Lustre: Skipped 1 previous similar message [ 1261.583552] Lustre: Skipped 1 previous similar message [ 1266.678600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1266.690123] Lustre: Skipped 1 previous similar message [ 1267.231709] Lustre: server umount lustre-OST0000 complete [ 1271.780198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.806577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.817407] Lustre: Skipped 1 previous similar message [ 1276.898565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1276.902987] Lustre: Skipped 1 previous similar message [ 1277.715578] Lustre: server umount lustre-MDT0000 complete [ 1282.028650] LustreError: 35039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1282.054611] LustreError: 35039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1282.461474] LustreError: 35020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531065 with bad export cookie 5090373549018786288 [ 1282.466365] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1282.474137] LustreError: 35020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1282.861528] Lustre: server umount lustre-MDT0001 complete [ 1295.881589] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 1298.637160] Key type lgssc unregistered [ 1298.866612] LNet: 38576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.882254] LNetError: 38576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.901230] LNet: Removed LNI 192.168.202.126@tcp [ 1299.729522] Key type .llcrypt unregistered [ 1299.731707] Key type ._llcrypt unregistered [ 1328.816332] Key type ._llcrypt registered [ 1328.818577] Key type .llcrypt registered [ 1328.939786] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 1330.569710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1330.594542] alg: No test for adler32 (adler32-zlib) [ 1331.713388] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 1332.056527] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 1333.760302] Key type lgssc registered [ 1335.723105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1346.727997] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1346.749486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.216165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.177105] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1357.799299] LustreError: 40002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.807666] LustreError: 40002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1362.918056] LustreError: 40003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.416848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.848492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1370.057870] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1377.889960] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1384.477166] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1392.754289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1393.307990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1401.200260] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1409.961208] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1417.912535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1427.264187] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.26@tcp (stopping) [ 1428.451824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.461022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1428.476371] Lustre: Skipped 2 previous similar messages [ 1428.496201] Lustre: Skipped 2 previous similar messages [ 1432.348634] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.26@tcp (stopping) [ 1433.598525] LustreError: 40003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1433.611884] LustreError: 40003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1433.692635] Lustre: server umount lustre-MDT0000 complete [ 1437.482817] LustreError: 41498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.327757] LustreError: 40737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531221 with bad export cookie 8811920549533485608 [ 1438.335695] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1438.348315] LustreError: 40737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1438.693030] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1438.719093] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1442.596082] LustreError: 39999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1442.612250] LustreError: 39999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1443.810916] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1443.834542] Lustre: Skipped 1 previous similar message [ 1444.867555] Lustre: server umount lustre-MDT0001 complete [ 1466.995636] Lustre: server umount lustre-OST0000 complete [ 1486.749671] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 1490.241236] Key type lgssc unregistered [ 1490.627728] LNet: 43881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.638700] LNetError: 43881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1490.647979] LNet: Removed LNI 192.168.202.126@tcp [ 1491.617899] Key type .llcrypt unregistered [ 1491.619470] Key type ._llcrypt unregistered [ 1517.511871] Key type ._llcrypt registered [ 1517.513758] Key type .llcrypt registered [ 1517.584574] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 1518.751602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1518.788088] alg: No test for adler32 (adler32-zlib) [ 1519.906821] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 1520.156035] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 1521.881609] Key type lgssc registered [ 1523.059536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1531.647739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1531.661607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.188579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1538.420939] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1542.653662] LustreError: 45295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1542.680344] LustreError: 45295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1547.756677] LustreError: 45296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.713474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.174160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1554.808484] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1563.688495] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1570.209980] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1578.436418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1578.729088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.615835] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1596.224144] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1602.302928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1607.479414] Lustre: lustre-OST0000: Not available for connect from 192.168.202.26@tcp (stopping) [ 1607.651084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1607.662678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1609.729505] Lustre: server umount lustre-OST0000 complete [ 1631.215871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1631.228745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.250960] Lustre: Skipped 1 previous similar message [ 1631.266548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1631.289473] Lustre: Skipped 2 previous similar messages [ 1633.251155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1633.252306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1635.519078] Lustre: server umount lustre-MDT0000 complete [ 1638.370975] LustreError: 46025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.401617] LustreError: 46025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1640.187241] LustreError: 45278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531422 with bad export cookie 13778423466246619825 [ 1640.216347] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1647.245346] Lustre: server umount lustre-MDT0001 complete [ 1661.015855] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 1665.199457] Key type lgssc unregistered [ 1665.627787] LNet: 49004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.637829] LNetError: 49004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1665.663906] LNet: Removed LNI 192.168.202.126@tcp [ 1667.016912] Key type .llcrypt unregistered [ 1667.023772] Key type ._llcrypt unregistered [ 1693.952525] Key type ._llcrypt registered [ 1693.956375] Key type .llcrypt registered [ 1694.130824] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 1695.614180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1695.648778] alg: No test for adler32 (adler32-zlib) [ 1697.296859] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 1697.805226] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 1699.649602] Key type lgssc registered [ 1701.467969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1710.967981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1711.001588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1712.797832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.828441] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1722.341585] LustreError: 50417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.352504] LustreError: 50417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1727.457518] LustreError: 50418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1728.642035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.226662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1734.541449] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1743.803602] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1751.670683] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1760.374370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1760.855877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.104822] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1776.724920] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1782.449264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1788.939628] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1793.819116] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1806.408777] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0410h19m46s [ 1808.234673] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1810.920434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1810.933728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.961658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1812.005158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1812.016909] Lustre: Skipped 4 previous similar messages [ 1814.320427] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.26@tcp (stopping) [ 1816.962934] Lustre: server umount lustre-MDT0000 complete [ 1820.044836] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1821.475209] LustreError: 50414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1827.599515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.640578] LustreError: 50412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1827.674541] LustreError: 50412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1827.872538] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1828.299836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1828.382256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1829.360550] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1833.851695] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1838.334814] Lustre: DEBUG MARKER: Wait for df (36712) ... [ 1840.535760] Lustre: DEBUG MARKER: done [ 1846.242955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1846.253983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.268302] Lustre: Skipped 2 previous similar messages [ 1846.280053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1846.285554] Lustre: Skipped 4 previous similar messages [ 1848.804136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1848.821575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.726289] Lustre: server umount lustre-OST0000 complete [ 1860.072962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1860.076469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1860.087534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1860.103960] Lustre: Skipped 4 previous similar messages [ 1863.324053] Lustre: server umount lustre-MDT0000 complete [ 1867.242360] LustreError: 51145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1867.266351] LustreError: 51145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1868.028307] LustreError: 51152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531650 with bad export cookie 9883134822814747614 [ 1868.036023] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1868.039381] LustreError: 51152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1868.611188] Lustre: server umount lustre-MDT0001 complete [ 1882.050555] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 1885.578534] Key type lgssc unregistered [ 1885.994666] LNet: 55359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1886.002087] LNetError: 55359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1886.020980] LNet: Removed LNI 192.168.202.126@tcp [ 1887.299316] Key type .llcrypt unregistered [ 1887.301875] Key type ._llcrypt unregistered [ 1928.865340] Key type ._llcrypt registered [ 1928.867983] Key type .llcrypt registered [ 1928.977267] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 1929.976933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1930.133265] alg: No test for adler32 (adler32-zlib) [ 1931.254648] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 1931.441945] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 1933.096264] Key type lgssc registered [ 1934.558720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1942.236473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1942.249566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.564728] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1943.586981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1943.785350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1948.551608] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1956.958940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.030694] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1957.062486] Lustre: 57563:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1957.322305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1962.751600] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1970.734218] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1977.006519] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1984.642677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1984.862963] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1985.113388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.002345] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1994.500186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1998.992975] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2005.326694] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0410h23m06s [ 2006.757578] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2014.135402] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2016.724670] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2016.982239] LustreError: 57571:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2037.076077] Lustre: lustre-MDT0000: Client d26c3ab3-84a3-48da-ba8a-37ea1c905afe (at 192.168.202.26@tcp) reconnecting [ 2057.048516] LustreError: 57571:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2059.488824] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2062.516206] Lustre: DEBUG MARKER: done [ 2071.523040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2071.531509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.537019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2071.540095] Lustre: Skipped 1 previous similar message [ 2071.552237] Lustre: Skipped 1 previous similar message [ 2076.643161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2076.657421] Lustre: Skipped 1 previous similar message [ 2081.763763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2081.770934] Lustre: Skipped 1 previous similar message [ 2083.297540] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2083.485738] Lustre: server umount lustre-OST0000 complete [ 2090.475496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2090.498425] LustreError: Skipped 1 previous similar message [ 2090.507590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.538169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2093.903384] Lustre: server umount lustre-MDT0000 complete [ 2098.224055] LustreError: 57557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531881 with bad export cookie 14791983034313640285 [ 2098.229100] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2098.231984] LustreError: 57557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2098.536718] Lustre: server umount lustre-MDT0001 complete [ 2110.624580] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 2113.641961] Key type lgssc unregistered [ 2113.930793] LNet: 61894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.945409] LNetError: 61894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.975743] LNet: Removed LNI 192.168.202.126@tcp [ 2114.965419] Key type .llcrypt unregistered [ 2114.967810] Key type ._llcrypt unregistered [ 2166.508251] Key type ._llcrypt registered [ 2166.513193] Key type .llcrypt registered [ 2166.705050] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 2168.677815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2168.711112] alg: No test for adler32 (adler32-zlib) [ 2169.944534] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 2170.252464] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 2172.025035] Key type lgssc registered [ 2173.116263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2176.849794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2176.909234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.099822] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2178.118331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2178.243239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2179.203292] Lustre: Failing over lustre-MDT0000 [ 2179.848720] Lustre: server umount lustre-MDT0000 complete [ 2188.266570] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:26:09 (1788531969) [ 2195.729283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2196.142461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.917996] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2211.812155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.928418] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2211.965152] Lustre: 64700:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2212.296465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2217.383064] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2225.428638] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2231.918931] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2239.900557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2240.181575] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2240.487565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2247.419528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2247.771975] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2257.253698] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2264.229378] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2273.256758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.258298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2273.272582] Lustre: Skipped 2 previous similar messages [ 2273.283518] Lustre: Skipped 2 previous similar messages [ 2277.381894] Lustre: server umount lustre-MDT0000 complete [ 2282.023390] LustreError: 64690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532064 with bad export cookie 7473905504827784207 [ 2282.031416] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2282.038913] LustreError: 64690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2282.445468] Lustre: server umount lustre-MDT0001 complete [ 2286.435136] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2291.373163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2299.875804] Lustre: 63618:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788532066/real 1788532066] req@00000000d3213cbc x1875411681193216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788532082 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2299.926823] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.684420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.970146] Lustre: 63619:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788532070/real 1788532070] req@00000000e1e6f487 x1875411681193344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788532086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2305.056272] Lustre: 63619:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788532071/real 1788532071] req@00000000d422284b x1875411681193472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788532087 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2307.105259] LustreError: 63616:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a4ff3222 x1875411681194240/t0(0) o250->MGC192.168.202.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2307.470658] LustreError: 66550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.498640] LustreError: 66550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2307.620507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2307.744925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2308.576445] Lustre: 63618:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788532074/real 1788532074] req@000000009cde6c0c x1875411681193728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788532090 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2309.602138] LustreError: 66551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.628910] LustreError: 68471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.641133] LustreError: 68471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2312.673829] Lustre: 63617:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788532079/real 1788532079] req@0000000024a9b32a x1875411681193856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788532095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2313.150626] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2315.755031] LustreError: 68471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.775467] LustreError: 68471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2320.872370] LustreError: 68470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.899963] LustreError: 68470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2322.541176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.821963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2327.172767] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2328.055619] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2329.153554] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2335.519665] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2341.115401] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2349.538491] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2349.548392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2349.564281] Lustre: Skipped 1 previous similar message [ 2349.585058] Lustre: Skipped 2 previous similar messages [ 2354.662556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2354.681961] Lustre: Skipped 2 previous similar messages [ 2354.969056] Lustre: server umount lustre-MDT0000 complete [ 2359.358282] LustreError: 64692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532142 with bad export cookie 7473905504827786524 [ 2359.365641] LustreError: 64692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2359.366612] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2359.633511] Lustre: server umount lustre-MDT0001 complete [ 2366.979700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2376.160115] Lustre: 63618:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788532142/real 1788532142] req@000000005222ad2e x1875411681224704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788532158 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2376.194908] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.215753] Lustre: Skipped 1 previous similar message [ 2378.373214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.420317] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x67b8a5754c18cab7 [ 2384.428656] Lustre: MGC192.168.202.126@tcp: Connection restored to 0@lo (at 0@lo) [ 2384.787983] LustreError: 66550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.860448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.908056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2389.771305] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2397.620062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.618300] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2403.312103] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2410.059978] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2415.668798] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2423.778158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2423.785463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.807232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2427.891311] Lustre: server umount lustre-MDT0000 complete [ 2429.923743] LustreError: 71153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.954020] LustreError: 71153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2431.060461] LustreError: 64690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532213 with bad export cookie 7473905504827787959 [ 2431.066583] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2431.079538] LustreError: 64690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2431.676546] Lustre: server umount lustre-MDT0001 complete [ 2439.409559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2443.929704] Lustre: DEBUG MARKER: files compared the same [ 2457.149764] Lustre: server umount lustre-OST0000 complete [ 2477.387650] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 2480.816726] Key type lgssc unregistered [ 2481.196917] LNet: 74429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2481.208471] LNetError: 74429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2481.229740] LNet: Removed LNI 192.168.202.126@tcp [ 2482.244735] Key type .llcrypt unregistered [ 2482.251676] Key type ._llcrypt unregistered [ 2506.357938] Key type ._llcrypt registered [ 2506.359920] Key type .llcrypt registered [ 2506.461396] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 2507.883674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2507.929665] alg: No test for adler32 (adler32-zlib) [ 2509.082191] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 2509.312304] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 2510.984390] Key type lgssc registered [ 2512.259898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2520.566453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2520.590109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.102745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.676465] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug +malloc all [ 2531.304115] LustreError: 75844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.336721] LustreError: 75844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2536.185293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.456496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2541.002880] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug +malloc all [ 2548.722887] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2555.082692] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2562.954371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2563.336779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2566.454624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2570.394778] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug +malloc all [ 2578.083819] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2592.225323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.227231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2592.244979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2595.170770] Lustre: server umount lustre-OST0000 complete [ 2602.464852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.471449] Lustre: Skipped 1 previous similar message [ 2602.474436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2602.477188] Lustre: Skipped 1 previous similar message [ 2606.331988] Lustre: server umount lustre-MDT0000 complete [ 2607.586342] LustreError: 75844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.617228] LustreError: 75844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2611.312275] LustreError: 75826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532394 with bad export cookie 3537569733835347283 [ 2611.319542] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2611.327344] LustreError: 75826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2611.688207] Lustre: server umount lustre-MDT0001 complete [ 2623.425203] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 2625.961801] Key type lgssc unregistered [ 2626.307834] LNet: 79377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2626.316420] LNetError: 79377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2626.332631] LNet: Removed LNI 192.168.202.126@tcp [ 2627.204518] Key type .llcrypt unregistered [ 2627.206645] Key type ._llcrypt unregistered [ 2650.284110] Key type ._llcrypt registered [ 2650.285811] Key type .llcrypt registered [ 2650.403523] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 2651.480122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2651.543688] alg: No test for adler32 (adler32-zlib) [ 2652.683873] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 2653.089289] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 2654.888248] Key type lgssc registered [ 2656.749883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2666.955612] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2666.968299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2668.079172] Lustre: 80778:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2683.680311] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2688.800340] LustreError: 80765:0:(mgc_request.c:317:do_config_log_add()) MGC192.168.202.126@tcp: failed processing log, type 4: rc = -110 [ 2693.603146] Lustre: 80780:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2699.104327] Lustre: 80765:0:(mgc_request.c:2298:mgc_process_log()) MGC192.168.202.126@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 2709.369133] Lustre: 80778:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2709.472365] Lustre: 80782:0:(mgc_request.c:2298:mgc_process_log()) MGC192.168.202.126@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 2714.594538] Lustre: 80780:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2719.648546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2719.719330] LustreError: 80782:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 2719.722619] Lustre: 80778:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2727.617451] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2729.954340] Lustre: 80778:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2729.970597] Lustre: 80778:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [ 2736.099714] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2740.258721] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2743.495021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.614602] Lustre: *** cfs_fail_loc=706, val=0*** [ 2744.077226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2744.132850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2748.744883] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2749.414923] LustreError: 80796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.435702] LustreError: 80796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2754.555972] LustreError: 81861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.584786] LustreError: 81861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2757.836457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2758.164034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2763.187693] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2771.366933] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2777.933960] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2783.718177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2783.729205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2783.737662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2783.753488] Lustre: Skipped 1 previous similar message [ 2783.767663] Lustre: Skipped 1 previous similar message [ 2786.970475] Lustre: server umount lustre-OST0000 complete [ 2793.953099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2793.957211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.960555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2793.974172] Lustre: Skipped 1 previous similar message [ 2799.076069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2799.084259] Lustre: Skipped 2 previous similar messages [ 2804.201727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2804.217282] Lustre: Skipped 1 previous similar message [ 2805.216727] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2805.329909] Lustre: server umount lustre-MDT0000 complete [ 2809.337786] LustreError: 81864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.362482] LustreError: 81864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2809.527402] LustreError: 80773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532592 with bad export cookie 1103581512759015968 [ 2809.551896] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2809.941903] Lustre: server umount lustre-MDT0001 complete [ 2820.656776] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 2823.390946] Key type lgssc unregistered [ 2823.628340] LNet: 84392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2823.638320] LNetError: 84392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2823.666173] LNet: Removed LNI 192.168.202.126@tcp [ 2824.277183] Key type .llcrypt unregistered [ 2824.284598] Key type ._llcrypt unregistered [ 2850.234962] Key type ._llcrypt registered [ 2850.237494] Key type .llcrypt registered [ 2850.356617] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 2851.753953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2851.851282] alg: No test for adler32 (adler32-zlib) [ 2853.014574] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 2853.284745] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 2854.945556] Key type lgssc registered [ 2856.183681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2863.792860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2863.830881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2869.599893] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2877.078274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.618445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2882.838559] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2883.052615] LustreError: 86493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.077827] LustreError: 86493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2888.167530] LustreError: 86492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.182047] LustreError: 86492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2891.315586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2891.788818] LustreError: 86493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2891.825414] LustreError: 86493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2891.825957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.875930] LustreError: 86497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2896.899384] LustreError: 86497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2898.743684] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2901.989200] LustreError: 86491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2902.019746] LustreError: 86491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2908.739980] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2908.761352] Lustre: MGS: Client f25eb3a8-9492-4ae1-b050-c407cde8b07a (at 0@lo) reconnecting [ 2908.797142] LustreError: 86507:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2909.104546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2909.244937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2915.450129] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2931.170087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2931.181221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2931.200063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2934.763173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2934.789856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2934.798781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2937.470642] Lustre: server umount lustre-OST0000 complete [ 2944.993187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.995567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2945.008117] Lustre: Skipped 1 previous similar message [ 2945.011495] Lustre: Skipped 1 previous similar message [ 2945.022782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2948.490977] Lustre: server umount lustre-MDT0000 complete [ 2950.119178] LustreError: 88238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.152663] LustreError: 88238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2952.865876] Lustre: server umount lustre-MDT0001 complete [ 2957.413926] Lustre: server umount MGS complete [ 2969.619841] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:39:10 (1788532750) [ 3000.037860] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 3009.280250] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 3045.200358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.564769] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3061.712351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.137262] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3062.244752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3062.456344] Lustre: lustre-MDT0001: new disk, initializing [ 3062.629764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3062.730275] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3062.748966] Lustre: 93683:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3062.798941] LustreError: 93683:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3069.051283] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3081.381395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3081.811657] Lustre: lustre-OST0000: new disk, initializing [ 3081.821135] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3081.834921] Lustre: 94450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3081.934961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.643898] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3092.513202] Lustre: 94907:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ef7b5fba x1875412397374336/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788532891 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3100.440411] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3108.079829] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3108.086876] Lustre: MGS: Client e6499d01-cfdc-4168-9eda-5b5dafdad04f (at 0@lo) reconnecting [ 3108.140958] Lustre: 92951:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3108.173101] LustreError: 93669:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3108.188224] LustreError: 93669:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3108.220928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3108.370939] Lustre: lustre-MDT0000: new disk, initializing [ 3108.692948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3108.761191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3108.793664] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3109.870901] Lustre: 95520:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000009f59fd41 x1875412397381632/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788532908 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3113.823820] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3115.127261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3115.136299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3115.170048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3119.083694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 3119.090364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3146.220071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3146.228395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3146.238754] Lustre: Skipped 1 previous similar message [ 3146.242923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3146.271245] Lustre: Skipped 2 previous similar messages [ 3151.331188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3151.340649] Lustre: Skipped 1 previous similar message [ 3160.033276] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3160.197696] Lustre: server umount lustre-OST0000 complete [ 3166.701851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3166.718567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3166.727133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3166.758924] Lustre: Skipped 1 previous similar message [ 3166.776129] Lustre: Skipped 2 previous similar messages [ 3170.449167] Lustre: server umount lustre-MDT0000 complete [ 3171.815846] LustreError: 93654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3171.836564] LustreError: 93654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3180.895963] Lustre: server umount lustre-MDT0001 complete [ 3185.013978] Lustre: server umount MGS complete [ 3197.986981] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:42:58 (1788532978) [ 3217.281546] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 3220.046502] Key type lgssc unregistered [ 3220.350674] LNet: 98021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3220.360634] LNetError: 98021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3220.376817] LNet: Removed LNI 192.168.202.126@tcp [ 3221.279790] Key type .llcrypt unregistered [ 3221.283470] Key type ._llcrypt unregistered [ 3241.709759] Key type ._llcrypt registered [ 3241.711469] Key type .llcrypt registered [ 3241.880824] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 3242.690805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3242.774219] alg: No test for adler32 (adler32-zlib) [ 3244.063958] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 3244.307646] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 3246.024253] Key type lgssc registered [ 3247.505465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3257.367785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3257.407699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3258.974835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.668742] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3269.091897] LustreError: 99242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.116426] LustreError: 99242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3274.115657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3274.211315] LustreError: 99243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3274.716047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3280.385687] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3289.839351] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3298.090097] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3306.006032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3306.423783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.292973] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3322.353183] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3328.964263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3344.353389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3344.356057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3344.401708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3344.417329] Lustre: Skipped 1 previous similar message [ 3347.630626] Lustre: server umount lustre-OST0000 complete [ 3354.598644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.600568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3354.633087] Lustre: Skipped 2 previous similar messages [ 3354.640716] Lustre: Skipped 1 previous similar message [ 3359.718874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3359.732183] Lustre: Skipped 1 previous similar message [ 3364.834449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3364.852260] Lustre: Skipped 1 previous similar message [ 3366.368083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3366.821232] Lustre: server umount lustre-MDT0000 complete [ 3369.963735] LustreError: 100407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3369.995463] LustreError: 100407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3371.811485] LustreError: 99977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533154 with bad export cookie 17756835349273716730 [ 3371.817790] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3371.827246] LustreError: 99977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3372.531538] Lustre: server umount lustre-MDT0001 complete [ 3385.302069] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 3389.350913] Key type lgssc unregistered [ 3389.867842] LNet: 102812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3389.874654] LNetError: 102812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3389.898395] LNet: Removed LNI 192.168.202.126@tcp [ 3391.185498] Key type .llcrypt unregistered [ 3391.190638] Key type ._llcrypt unregistered [ 3409.869745] Key type ._llcrypt registered [ 3409.872358] Key type .llcrypt registered [ 3410.098472] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 3411.557107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3411.819897] alg: No test for adler32 (adler32-zlib) [ 3413.151984] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 3413.473695] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 3415.232283] Key type lgssc registered [ 3416.564530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3424.018612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3424.027701] LustreError: 103893:0:(tgt_mount.c:2652:server_fill_super()) cfs_race id 716 sleeping [ 3424.033572] LustreError: 103894:0:(tgt_mount.c:2652:server_fill_super()) cfs_fail_race id 716 waking [ 3424.038585] LustreError: 103893:0:(tgt_mount.c:2652:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 3424.044164] LustreError: 103893:0:(tgt_mount.c:2664:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3424.048446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3424.049743] LustreError: 103893:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3425.621608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3435.547992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.031198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3441.433774] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3448.303023] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing lsmod [ 3451.824408] LustreError: 105089:0:(tgt_mount.c:2652:server_fill_super()) cfs_race id 716 sleeping [ 3451.896973] LustreError: 105103:0:(tgt_mount.c:2652:server_fill_super()) cfs_fail_race id 716 waking [ 3451.902847] LustreError: 105089:0:(tgt_mount.c:2652:server_fill_super()) cfs_fail_race id 716 awake: rc=4931 [ 3451.915826] LustreError: 105089:0:(tgt_mount.c:2664:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3451.920893] LustreError: 105089:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3451.927180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3452.462115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3456.488934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3456.501204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3456.522120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3457.506449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3457.508616] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3457.516408] Lustre: Skipped 1 previous similar message [ 3457.535482] Lustre: Skipped 1 previous similar message [ 3460.643403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3460.652111] Lustre: Skipped 3 previous similar messages [ 3465.697418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3465.717505] Lustre: Skipped 2 previous similar messages [ 3470.304569] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3470.665299] Lustre: server umount lustre-MDT0000 complete [ 3470.825508] LustreError: 103928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.835919] LustreError: 103928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3475.089984] LustreError: 103906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533257 with bad export cookie 13655935344623346648 [ 3475.105885] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3475.937477] LustreError: 105287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.941405] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3475.986511] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3477.088313] LustreError: 104362:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000091c8cc1e x1875412984554496/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3477.278494] Lustre: server umount lustre-MDT0001 complete [ 3481.599997] Lustre: server umount lustre-OST0000 complete [ 3491.607335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.407380] LustreError: 106165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3492.524431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3497.961825] LustreError: 106166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3498.721574] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3503.078806] LustreError: 106165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3510.365524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.922824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3516.848880] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3524.963946] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3532.396437] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3540.730776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3541.321336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3548.537795] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3557.523492] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3562.866814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3571.169459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3571.183682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3571.199144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3573.217155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3576.995703] Lustre: server umount lustre-OST0000 complete [ 3583.457865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3583.487900] Lustre: Skipped 2 previous similar messages [ 3588.592765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3588.595708] Lustre: Skipped 4 previous similar messages [ 3597.280257] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3597.573101] Lustre: server umount lustre-MDT0000 complete [ 3598.819464] LustreError: 106166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3598.861897] LustreError: 106166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3602.472294] LustreError: 106147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533385 with bad export cookie 13655935344623347404 [ 3602.485745] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3602.497287] LustreError: 106147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 9 previous similar messages [ 3603.160457] Lustre: server umount lustre-MDT0001 complete [ 3617.376838] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 3621.135684] Key type lgssc unregistered [ 3621.522257] LNet: 109718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3621.536543] LNetError: 109718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3621.569766] LNet: Removed LNI 192.168.202.126@tcp [ 3622.630924] Key type .llcrypt unregistered [ 3622.634553] Key type ._llcrypt unregistered [ 3649.456515] Key type ._llcrypt registered [ 3649.460358] Key type .llcrypt registered [ 3649.547923] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 3651.076803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3651.155488] alg: No test for adler32 (adler32-zlib) [ 3652.283733] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 3652.641596] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 3654.432479] Key type lgssc registered [ 3655.873798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3665.760635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3665.782267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3667.173659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.381879] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3676.649207] LustreError: 111132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.676467] LustreError: 111132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3681.787782] LustreError: 111133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3683.045755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3683.693479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3689.998259] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3700.148844] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3708.501233] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3717.762408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3718.342481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3726.775497] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3736.616656] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3736.793039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3742.544485] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3747.095676] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3754.468520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3754.485554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.520615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3755.360860] Lustre: server umount lustre-OST0000 complete [ 3762.661385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.676409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3762.688017] Lustre: Skipped 1 previous similar message [ 3762.708054] Lustre: Skipped 1 previous similar message [ 3767.781363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3767.809419] Lustre: Skipped 1 previous similar message [ 3772.236126] Lustre: server umount lustre-MDT0000 complete [ 3772.899887] LustreError: 111132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3772.934032] LustreError: 111132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3778.223249] LustreError: 111111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533561 with bad export cookie 2293069332743356978 [ 3778.236247] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3778.254922] LustreError: 111111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3779.188301] Lustre: server umount lustre-MDT0001 complete [ 3797.062907] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 3801.362372] Key type lgssc unregistered [ 3802.121594] LNet: 114772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3802.156353] LNetError: 114772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3802.187918] LNet: Removed LNI 192.168.202.126@tcp [ 3803.707085] Key type .llcrypt unregistered [ 3803.709698] Key type ._llcrypt unregistered [ 3828.169716] Key type ._llcrypt registered [ 3828.172787] Key type .llcrypt registered [ 3828.382393] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 3830.020645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3830.059954] alg: No test for adler32 (adler32-zlib) [ 3831.143234] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 3831.532750] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 3833.336263] Key type lgssc registered [ 3834.898423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3845.173870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3845.193980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3846.643041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3852.336264] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3856.871231] LustreError: 115995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3856.883654] LustreError: 115995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3862.007501] LustreError: 115994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3862.526731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3862.862256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3868.503651] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3878.915625] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3886.377585] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3895.243199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3895.762908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3903.717843] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3912.694713] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3917.307997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3928.545664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3928.562727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.584420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3932.643568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3932.655827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3932.663488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.692446] Lustre: Skipped 1 previous similar message [ 3933.859360] Lustre: server umount lustre-OST0000 complete [ 3939.810891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3939.840375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3939.887988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3942.882831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.884659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3945.189000] Lustre: server umount lustre-MDT0000 complete [ 3948.008086] LustreError: 115994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3948.032036] LustreError: 115994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3949.811818] LustreError: 115973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533732 with bad export cookie 15992630679694089155 [ 3949.827200] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3950.490888] Lustre: server umount lustre-MDT0001 complete [ 3965.509058] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 3969.728873] Key type lgssc unregistered [ 3970.149394] LNet: 119554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3970.175411] LNetError: 119554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3970.223418] LNet: Removed LNI 192.168.202.126@tcp [ 3971.535743] Key type .llcrypt unregistered [ 3971.537571] Key type ._llcrypt unregistered [ 4000.636812] Key type ._llcrypt registered [ 4000.639946] Key type .llcrypt registered [ 4000.806583] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 4002.358369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4002.407050] alg: No test for adler32 (adler32-zlib) [ 4003.447184] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 4003.787220] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 4005.536497] Key type lgssc registered [ 4007.321627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4017.321523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4017.335910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4019.333162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4025.302394] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4029.928472] LustreError: 120969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4029.950341] LustreError: 120969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4035.046031] LustreError: 120970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.120634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.628015] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4041.122408] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4048.602964] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4054.827842] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4062.398080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4062.760233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4064.860936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 4069.469768] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4078.567689] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4085.377593] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4089.514767] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4089.521378] Lustre: Skipped 2 previous similar messages [ 4094.161733] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4094.204986] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4094.484857] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4094.489048] Lustre: Skipped 2 previous similar messages [ 4100.217319] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4100.219995] Lustre: Skipped 1 previous similar message [ 4106.901092] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.26@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4106.915270] Lustre: Skipped 2 previous similar messages [ 4107.038145] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.202.26@tcp 192.168.0.[2,10]@tcp [ 4114.687675] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.26@tcp 192.168.0.[2,10]@tcp [ 4118.736215] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.26/24@tcp in log lustre-MDT0000 [ 4118.740680] Lustre: Skipped 2 previous similar messages [ 4124.791793] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.26/24@tcp [ 4131.814061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4131.824082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4131.833502] Lustre: Skipped 1 previous similar message [ 4131.834228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4131.858363] Lustre: Skipped 1 previous similar message [ 4133.420433] Lustre: server umount lustre-OST0000 complete [ 4137.689129] Lustre: server umount lustre-MDT0000 complete [ 4137.953626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4137.963240] LustreError: Skipped 1 previous similar message [ 4137.973692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.990988] LustreError: 120969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.908951] LustreError: 120950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533924 with bad export cookie 12081739234377132547 [ 4141.920085] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4141.920976] LustreError: 120950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4144.321058] Lustre: server umount lustre-MDT0001 complete [ 4156.829563] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 4160.376850] Key type lgssc unregistered [ 4160.721665] LNet: 125819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4160.725951] LNetError: 125819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4160.737099] LNet: Removed LNI 192.168.202.126@tcp [ 4161.582357] Key type .llcrypt unregistered [ 4161.584230] Key type ._llcrypt unregistered [ 4201.586296] Key type ._llcrypt registered [ 4201.593949] Key type .llcrypt registered [ 4201.767992] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 4203.694271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4203.737228] alg: No test for adler32 (adler32-zlib) [ 4204.960826] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 4205.374314] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 4207.154109] Key type lgssc registered [ 4208.468744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4218.171760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4218.218909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.770129] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4219.815989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4219.844842] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.26/24@tcp [ 4226.051954] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4230.130431] LustreError: 127383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4230.150055] LustreError: 127383:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4235.244947] LustreError: 127384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.287282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.930980] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4237.018671] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.26/24@tcp [ 4237.069703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4242.988364] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4252.011326] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4260.061969] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4269.275398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4270.202552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4278.671976] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4288.931427] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4289.658118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 4300.276077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4300.285748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4300.297501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4300.313612] Lustre: Skipped 1 previous similar message [ 4300.335903] Lustre: Skipped 1 previous similar message [ 4301.965880] Lustre: server umount lustre-OST0000 complete [ 4308.961067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4308.965073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4308.972654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4310.499279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.500359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4310.529420] Lustre: Skipped 1 previous similar message [ 4313.000621] Lustre: server umount lustre-MDT0000 complete [ 4315.618806] LustreError: 128680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4317.567188] LustreError: 127362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534100 with bad export cookie 17433766009426337225 [ 4317.581845] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4317.593919] LustreError: 127362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4318.423939] Lustre: server umount lustre-MDT0001 complete [ 4331.805685] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 4334.691655] Key type lgssc unregistered [ 4335.143062] LNet: 130985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4335.150461] LNetError: 130985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4336.179290] LNet: Removed LNI 192.168.202.126@tcp [ 4337.028338] Key type .llcrypt unregistered [ 4337.030238] Key type ._llcrypt unregistered [ 4373.765032] Key type ._llcrypt registered [ 4373.769397] Key type .llcrypt registered [ 4373.846508] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 4391.188598] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 4392.794854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4392.891655] alg: No test for adler32 (adler32-zlib) [ 4393.885974] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 4394.111124] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 4395.793314] Key type lgssc registered [ 4396.855753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4443.052378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4443.084489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4444.391175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4444.436932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4444.593295] Lustre: lustre-MDT0000: new disk, initializing [ 4444.668645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4444.694205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4449.663797] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4462.156794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4462.315764] Lustre: 134386:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4462.497550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4462.506880] Lustre: Skipped 1 previous similar message [ 4462.609223] Lustre: lustre-MDT0001: new disk, initializing [ 4462.720306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4462.771802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4462.800996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4469.517210] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4482.099770] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4489.150293] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4497.853898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4498.392979] Lustre: lustre-OST0000: new disk, initializing [ 4498.399190] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4498.426317] Lustre: 136356:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4498.568574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4505.501140] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4507.319182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4507.335442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4507.454189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4516.394453] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4522.058773] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4522.334188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4527.144684] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4527.297362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4533.223242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4533.230443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4533.231517] LustreError: Skipped 1 previous similar message [ 4533.244312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4533.262928] Lustre: Skipped 1 previous similar message [ 4533.301408] Lustre: Skipped 1 previous similar message [ 4535.882638] Lustre: server umount lustre-OST0000 complete [ 4543.458265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.478883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4543.483457] Lustre: Skipped 1 previous similar message [ 4546.471637] Lustre: server umount lustre-MDT0000 complete [ 4550.169283] LustreError: 134377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534332 with bad export cookie 8844967364899428997 [ 4550.174195] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4550.181430] LustreError: 134377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4550.522475] Lustre: server umount lustre-MDT0001 complete [ 4560.797232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4561.563147] LustreError: 138573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4561.595046] LustreError: 138573:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4561.714790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.014940] LustreError: 138574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4567.109858] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4572.150089] LustreError: 138573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4577.256946] LustreError: 138574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4577.379933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4577.822994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4582.992418] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4592.638702] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4599.468180] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4606.703294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4607.023718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4613.576138] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4622.457794] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4627.490117] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4631.931417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4636.733763] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4636.980924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4645.729041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4646.027419] Lustre: lustre-OST0001: new disk, initializing [ 4646.042882] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4646.058582] Lustre: 141975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4646.187343] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4651.777574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4651.785793] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4651.989948] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4653.617461] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4663.081285] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4663.326704] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4668.108220] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4668.329363] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4675.529241] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff907e9816d000.ost_server_uuid 50 [ 4677.281862] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff907e9816d000.ost_server_uuid in IDLE state after 0 sec [ 4687.858874] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4687.862976] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4687.869970] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4687.874414] Lustre: Skipped 2 previous similar messages [ 4687.892993] Lustre: Skipped 1 previous similar message [ 4691.923404] Lustre: server umount lustre-OST0001 complete [ 4692.974327] LustreError: 140398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4692.991722] LustreError: 140398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4698.084869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4698.134032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4698.144139] Lustre: Skipped 1 previous similar message [ 4701.545749] Lustre: server umount lustre-OST0000 complete [ 4705.763194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4705.772740] LustreError: Skipped 1 previous similar message [ 4705.781491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4705.795798] Lustre: Skipped 1 previous similar message [ 4705.803788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4711.650967] Lustre: server umount lustre-MDT0000 complete [ 4713.444300] LustreError: 139696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4713.479754] LustreError: 139696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4715.738368] LustreError: 139308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534498 with bad export cookie 8844967364899429914 [ 4715.743973] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4715.744405] LustreError: 139308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4716.054555] Lustre: server umount lustre-MDT0001 complete [ 4737.199618] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 4740.160776] Key type lgssc unregistered [ 4740.409550] LNet: 144979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4740.425896] LNetError: 144979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4740.448402] LNet: Removed LNI 192.168.202.126@tcp [ 4741.112604] Key type .llcrypt unregistered [ 4741.114323] Key type ._llcrypt unregistered [ 4783.778356] Key type ._llcrypt registered [ 4783.788443] Key type .llcrypt registered [ 4784.016917] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 4801.808495] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 4803.197306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4803.335074] alg: No test for adler32 (adler32-zlib) [ 4804.686313] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 4805.129940] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 4806.872537] Key type lgssc registered [ 4808.377252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4847.245187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4847.279226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4848.547632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4848.589463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4848.660837] Lustre: lustre-MDT0000: new disk, initializing [ 4848.737487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4848.751264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4853.281793] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4863.416723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4863.533942] Lustre: 149131:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4863.606671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4863.610311] Lustre: Skipped 1 previous similar message [ 4863.665575] Lustre: lustre-MDT0001: new disk, initializing [ 4863.749097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4863.824069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4863.838669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4868.777951] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4878.340502] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4885.031438] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4891.504039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4891.858838] Lustre: lustre-OST0000: new disk, initializing [ 4891.861882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4891.869929] Lustre: 151102:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4891.951432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4898.052511] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4899.531168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4899.555293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4899.787676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4909.664884] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4914.473703] Lustre: Failing over lustre-OST0000 [ 4915.171111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4915.176994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4915.185274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4915.213154] Lustre: Skipped 1 previous similar message [ 4916.565084] Lustre: lustre-OST0000: Not available for connect from 192.168.202.26@tcp (stopping) [ 4916.581558] Lustre: Skipped 1 previous similar message [ 4916.814535] Lustre: server umount lustre-OST0000 complete [ 4936.786067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4937.205916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4937.241996] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4942.132977] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4944.845661] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4948.898840] Lustre: Failing over lustre-MDT0000 [ 4949.204141] Lustre: server umount lustre-MDT0000 complete [ 4949.323958] LustreError: MGC192.168.202.126@tcp: operation ldlm_enqueue to node 0@lo failed: rc = -107 [ 4949.326615] LustreError: 151101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534732 with bad export cookie 9668868003581226714 [ 4949.331470] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4949.331966] Lustre: 149135:0:(mgc_request.c:2298:mgc_process_log()) MGC192.168.202.126@tcp: IR log lustre-mdtir failed, not fatal: rc = -107 [ 4949.337391] Lustre: 152910:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4950.499307] Lustre: 152906:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4950.500750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4950.513919] LustreError: 149143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4950.513930] LustreError: 149143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4950.593825] Lustre: Skipped 1 previous similar message [ 4952.449567] LustreError: 149138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4952.476957] LustreError: 149138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4955.620485] LustreError: 150017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4955.621749] Lustre: 152909:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4955.645906] LustreError: 150017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4958.177938] Lustre: 152906:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4958.178341] LustreError: 149143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4958.195305] Lustre: 152906:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [ 4958.205802] LustreError: 149143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4962.586445] LustreError: 149139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4962.607898] LustreError: 149139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4966.371818] Lustre: 152906:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4966.397673] Lustre: 152906:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [ 4970.972122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4971.496257] LustreError: 149138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4971.548856] LustreError: 149138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4978.659140] Lustre: 149135:0:(mgc_request.c:2298:mgc_process_log()) MGC192.168.202.126@tcp: IR log lustre-mdtir failed, not fatal: rc = -108 [ 4988.189733] LustreError: 153335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4988.222400] LustreError: 153335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 4997.091429] LustreError: 147353:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fd4b420b x1875414443154944/t0(0) o250->MGC192.168.202.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4997.345754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4997.428103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4997.503528] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4998.432490] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5002.343566] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5002.741045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5002.763890] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5002.776408] Lustre: Skipped 1 previous similar message [ 5002.813526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5012.980183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5012.980810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5012.982580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5013.051858] Lustre: Skipped 2 previous similar messages [ 5015.995421] Lustre: server umount lustre-OST0000 complete [ 5023.208491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5023.211637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5023.221262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5023.228286] Lustre: Skipped 1 previous similar message [ 5023.255800] Lustre: Skipped 2 previous similar messages [ 5026.257036] Lustre: server umount lustre-MDT0000 complete [ 5028.328361] LustreError: 153335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5028.362550] LustreError: 153335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5030.937761] LustreError: 149123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534813 with bad export cookie 9668868003581228128 [ 5030.950783] LustreError: 149123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5030.953926] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5031.541533] Lustre: server umount lustre-MDT0001 complete [ 5045.135790] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 5049.033738] Key type lgssc unregistered [ 5049.426580] LNet: 154768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5049.434192] LNetError: 154768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5049.452962] LNet: Removed LNI 192.168.202.126@tcp [ 5050.447324] Key type .llcrypt unregistered [ 5050.456522] Key type ._llcrypt unregistered [ 5090.893896] Key type ._llcrypt registered [ 5090.896970] Key type .llcrypt registered [ 5091.065944] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 5092.531166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5092.661283] alg: No test for adler32 (adler32-zlib) [ 5093.998794] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 5094.289312] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 5095.984448] Key type lgssc registered [ 5097.593152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5106.930859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5106.947958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5108.544072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5114.766136] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5118.959685] LustreError: 156287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5118.981964] LustreError: 156287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5124.072734] LustreError: 156288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5125.566161] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5129.191660] LustreError: 156287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5130.626063] LustreError: 156316:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5130.648262] LustreError: 156316:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5130.656768] LustreError: 156316:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 5131.242251] Lustre: server umount lustre-MDT0000 complete [ 5151.016559] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 5163.159764] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 5210.449675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5210.745055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5210.778331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5210.912918] Lustre: lustre-MDT0000: new disk, initializing [ 5211.038924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5211.061265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5217.761963] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5231.434150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5231.638330] Lustre: 159622:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5231.752455] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5231.756656] Lustre: Skipped 1 previous similar message [ 5231.896836] Lustre: lustre-MDT0001: new disk, initializing [ 5232.027869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5232.072033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5232.087511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5237.170338] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5248.522981] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5255.435376] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5263.471846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5263.984320] Lustre: lustre-OST0000: new disk, initializing [ 5263.990507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5264.003462] Lustre: 161595:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5264.158086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5270.636490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5270.647120] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5270.802710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5272.153390] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5283.393683] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5287.572644] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5290.258732] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:17:50 (1788535070) [ 5578.209940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.213798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5578.234594] Lustre: Skipped 2 previous similar messages [ 5578.236983] Lustre: Skipped 1 previous similar message [ 5583.329101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5583.331209] Lustre: Skipped 2 previous similar messages [ 5584.110486] Lustre: server umount lustre-MDT0000 complete [ 5588.460800] LustreError: 159635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5588.496457] LustreError: 159635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5588.681109] LustreError: 159616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788535371 with bad export cookie 6879248044980832576 [ 5588.693611] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5588.697966] LustreError: 159616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5593.582233] LustreError: 159629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5593.584616] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5593.628326] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5598.696625] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5603.296255] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5603.701818] Lustre: server umount lustre-MDT0001 complete [ 5614.714460] Lustre: server umount lustre-OST0000 complete [ 5626.843619] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 5637.734581] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 5679.095905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5679.453572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5679.490272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5679.592412] Lustre: lustre-MDT0000: new disk, initializing [ 5679.770472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5679.809044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5685.628681] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5698.401441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.527817] Lustre: 166265:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5698.621911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5698.630300] Lustre: Skipped 1 previous similar message [ 5698.716490] Lustre: lustre-MDT0001: new disk, initializing [ 5698.897455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5698.956736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5698.978822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5703.920756] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5713.462888] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5719.386788] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5727.112648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5727.404607] Lustre: lustre-OST0000: new disk, initializing [ 5727.411838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5727.421831] Lustre: 168231:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5727.516687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5734.551210] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5737.614460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5737.623183] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5737.830927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5744.067554] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5748.720307] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5748.943735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5753.479935] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5753.789719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5758.444678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5758.446958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5758.456651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5758.456663] Lustre: Skipped 1 previous similar message [ 5758.508852] Lustre: Skipped 1 previous similar message [ 5762.085370] Lustre: server umount lustre-OST0000 complete [ 5768.673505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5768.681651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5768.688503] Lustre: Skipped 1 previous similar message [ 5768.712441] Lustre: Skipped 2 previous similar messages [ 5772.525847] Lustre: server umount lustre-MDT0000 complete [ 5773.797981] LustreError: 167393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5773.822838] LustreError: 167393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5776.527969] LustreError: 169570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788535559 with bad export cookie 6879248044980991448 [ 5776.529130] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5776.543756] LustreError: 169570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5776.888906] Lustre: server umount lustre-MDT0001 complete [ 5785.813701] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:26:07 (1788535567) [ 5804.438831] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 5815.244672] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 5859.311152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5859.814330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5859.867470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5859.968862] Lustre: lustre-MDT0000: new disk, initializing [ 5860.164107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.218836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5866.685504] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5878.722277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.863886] Lustre: 172994:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5879.269164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5884.372856] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5894.687910] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5901.679433] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5909.031490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5909.508646] Lustre: lustre-OST0000: new disk, initializing [ 5909.519327] Lustre: Skipped 1 previous similar message [ 5909.533697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5909.548157] Lustre: Skipped 2 previous similar messages [ 5909.551175] Lustre: 174974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5916.858608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5916.879112] Lustre: Skipped 1 previous similar message [ 5916.884386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5916.988283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5917.962440] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5928.953222] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5937.638682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5937.645965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5937.654679] Lustre: Skipped 1 previous similar message [ 5937.656694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5943.793980] Lustre: server umount lustre-OST0000 complete [ 5950.950340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5950.963449] LustreError: Skipped 1 previous similar message [ 5950.968786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5955.273585] Lustre: server umount lustre-MDT0000 complete [ 5958.114214] LustreError: 173001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5958.132960] LustreError: 173001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5960.225442] LustreError: 172986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788535743 with bad export cookie 6879248044980992365 [ 5960.227995] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5960.234920] LustreError: 172986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5960.673308] Lustre: server umount lustre-MDT0001 complete [ 5973.477939] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:29:14 (1788535754) [ 5994.942110] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 6007.404342] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 6051.504791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6051.910811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6051.985934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6052.140175] Lustre: lustre-MDT0000: new disk, initializing [ 6052.308607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6052.321571] Lustre: Skipped 2 previous similar messages [ 6052.388184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6059.233938] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6073.592228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6073.736671] Lustre: 179298:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6074.301090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6080.376113] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6091.722734] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6099.184329] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6110.722525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6111.232092] Lustre: 181277:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6119.028694] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6120.090741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6120.230616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6129.980202] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6140.897153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6140.902177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6140.902699] LustreError: Skipped 1 previous similar message [ 6140.903300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6140.903305] Lustre: Skipped 6 previous similar messages [ 6140.959245] Lustre: Skipped 2 previous similar messages [ 6145.420283] Lustre: server umount lustre-OST0000 complete [ 6151.139280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6156.470339] Lustre: server umount lustre-MDT0000 complete [ 6160.354558] LustreError: 180680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6160.372530] LustreError: 180680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6160.853248] LustreError: 179289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788535943 with bad export cookie 6879248044980993492 [ 6160.855890] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6160.871696] LustreError: 179289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6161.280144] Lustre: server umount lustre-MDT0001 complete [ 6173.003416] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 6175.410967] Key type lgssc unregistered [ 6175.651828] LNet: 183133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6175.657975] LNetError: 183133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6175.672651] LNet: Removed LNI 192.168.202.126@tcp [ 6176.304866] Key type .llcrypt unregistered [ 6176.308335] Key type ._llcrypt unregistered [ 6201.879408] Key type ._llcrypt registered [ 6201.882374] Key type .llcrypt registered [ 6201.990675] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 6203.613201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6203.696154] alg: No test for adler32 (adler32-zlib) [ 6204.824962] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 6205.146702] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 6206.896295] Key type lgssc registered [ 6208.304241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6217.972892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6218.005246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6219.729861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6225.420575] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6229.985328] LustreError: 184548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6229.995112] LustreError: 184548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6235.056204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.105420] LustreError: 184549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.738919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6240.620524] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6249.056668] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6255.497586] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6263.533560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6263.928715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6264.952189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6271.080285] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6279.606089] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6301.156250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6301.157440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6301.162189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6301.168040] Lustre: Skipped 1 previous similar message [ 6301.173292] LustreError: Skipped 1 previous similar message [ 6305.041201] Lustre: server umount lustre-OST0000 complete [ 6311.393609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6311.405056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6311.409325] Lustre: Skipped 1 previous similar message [ 6311.421805] Lustre: Skipped 2 previous similar messages [ 6315.574320] Lustre: server umount lustre-MDT0000 complete [ 6316.517968] LustreError: 187054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.539112] LustreError: 187054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6320.700141] LustreError: 184528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536103 with bad export cookie 5102274854086096550 [ 6320.702281] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6320.705300] LustreError: 184528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6321.004449] Lustre: server umount lustre-MDT0001 complete [ 6335.850415] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 6339.942335] Key type lgssc unregistered [ 6340.442607] LNet: 188111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6340.449928] LNetError: 188111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6341.485601] LNet: Removed LNI 192.168.202.126@tcp [ 6342.543040] Key type .llcrypt unregistered [ 6342.550349] Key type ._llcrypt unregistered [ 6370.369908] Key type ._llcrypt registered [ 6370.372751] Key type .llcrypt registered [ 6370.489971] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 6371.898608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6371.978562] alg: No test for adler32 (adler32-zlib) [ 6373.210571] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 6373.434302] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 6375.088202] Key type lgssc registered [ 6376.526483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6384.364839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6384.394381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6386.148885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6391.301717] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6395.366531] LustreError: 189524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6395.384023] LustreError: 189524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6399.786821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6400.151389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6404.736416] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6412.305732] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6418.155261] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6425.907964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6426.489584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6430.584854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6433.088479] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6440.381718] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6451.180271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6451.186199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6451.201775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6452.737530] Lustre: server umount lustre-OST0000 complete [ 6461.098542] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff907e87275000.ost_server_uuid 50 [ 6472.160208] Lustre: 188837:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788536238/real 1788536238] req@00000000b5569aae x1875416088342272/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788536254 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6472.178531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6473.157861] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6476.321278] Lustre: 188837:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788536243/real 1788536243] req@00000000e9c75bcf x1875416088343936/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788536259 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6481.376415] Lustre: 188836:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788536248/real 1788536248] req@00000000bd9625e2 x1875416088344960/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788536264 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6488.032339] Lustre: 188836:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788536253/real 1788536253] req@00000000cc47379d x1875416088345984/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788536269 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6491.619149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6491.626967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6491.649832] Lustre: Skipped 1 previous similar message [ 6491.661954] Lustre: Skipped 1 previous similar message [ 6494.802163] Lustre: server umount lustre-MDT0000 complete [ 6496.738158] LustreError: 189520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6496.784982] LustreError: 189520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6497.768595] LustreError: 189524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6497.794803] LustreError: 189524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6499.626432] LustreError: 189505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536282 with bad export cookie 17170922803776682631 [ 6499.628209] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6499.638614] LustreError: 189505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6500.128554] Lustre: server umount lustre-MDT0001 complete [ 6511.682052] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:38:12 (1788536292) [ 6519.540764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6520.353444] LustreError: 193467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.511562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6524.999377] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6525.931675] LustreError: 193468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6534.567087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.415746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6539.734892] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6547.780613] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6554.052951] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6562.785634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6563.192708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6566.264808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6571.098445] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6582.842407] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6591.728856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6592.148127] Lustre: lustre-OST0001: new disk, initializing [ 6592.150953] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6592.158632] Lustre: 196310:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6592.294100] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6598.051790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6598.065331] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6598.259430] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6601.555120] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6612.849157] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6616.545068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6616.554288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6616.575326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6617.571052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6618.596855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6618.611343] Lustre: Skipped 2 previous similar messages [ 6622.213638] Lustre: server umount lustre-OST0000 complete [ 6623.729919] LustreError: 195291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6623.764318] LustreError: 195291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6630.821847] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6631.101267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6638.177209] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6638.467705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6644.200209] LustreError: 195289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6644.210305] LustreError: 195289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6654.433938] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6654.445766] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6654.448726] LustreError: Skipped 1 previous similar message [ 6654.464723] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6654.465148] Lustre: Skipped 2 previous similar messages [ 6654.487780] Lustre: Skipped 2 previous similar messages [ 6659.265938] Lustre: server umount lustre-OST0001 complete [ 6663.138940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6663.146169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6663.158901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6669.180314] Lustre: server umount lustre-MDT0000 complete [ 6672.822422] LustreError: 193447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536455 with bad export cookie 17170922803776683758 [ 6672.836180] LustreError: 193447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6672.838076] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6673.188411] Lustre: server umount lustre-MDT0001 complete [ 6698.975623] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:41:20 (1788536480) [ 6705.315306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6705.602520] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6705.649881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6705.908958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6710.841233] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6720.010301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6720.060932] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6720.092676] Lustre: 199751:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6725.389371] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6733.424276] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6741.522907] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6751.557588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6751.852346] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6760.081737] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6761.309326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6768.585505] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6775.287455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6775.348739] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6775.747710] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6775.766876] Lustre: Skipped 2 previous similar messages [ 6781.839594] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6783.629864] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6791.682939] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6797.286359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6797.290229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6797.292673] LustreError: Skipped 1 previous similar message [ 6797.299874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6797.306730] Lustre: Skipped 2 previous similar messages [ 6797.330739] Lustre: Skipped 3 previous similar messages [ 6800.567553] Lustre: server umount lustre-OST0000 complete [ 6802.409180] LustreError: 203137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6802.440229] LustreError: 203137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6817.767209] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6817.785261] LustreError: Skipped 1 previous similar message [ 6822.590895] Lustre: server umount lustre-OST0001 complete [ 6832.923518] Lustre: server umount lustre-MDT0000 complete [ 6836.495884] LustreError: 201621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536619 with bad export cookie 17170922803776685291 [ 6836.504188] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6836.507508] LustreError: 201621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6836.847810] Lustre: server umount lustre-MDT0001 complete [ 6859.636558] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:44:01 (1788536641) [ 6877.931370] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 6891.232961] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 6930.628453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.836821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6930.884596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6930.945978] Lustre: lustre-MDT0000: new disk, initializing [ 6931.032261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6931.045324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6935.876350] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6946.891629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6947.043911] Lustre: 207849:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6947.141185] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6947.145442] Lustre: Skipped 1 previous similar message [ 6947.212443] Lustre: lustre-MDT0001: new disk, initializing [ 6947.353443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6947.373627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6951.379500] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6963.931641] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6970.060925] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6977.462636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6977.735619] Lustre: lustre-OST0000: new disk, initializing [ 6977.742456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6977.751721] Lustre: 209815:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6984.587531] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6987.876253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6987.883606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6988.042385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6993.509870] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6998.453529] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6998.649444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7003.161865] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7003.422093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7008.738350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7008.739238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7008.749036] LustreError: Skipped 2 previous similar messages [ 7008.767915] Lustre: Skipped 4 previous similar messages [ 7008.785930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7008.785941] Lustre: Skipped 5 previous similar messages [ 7011.579068] Lustre: server umount lustre-OST0000 complete [ 7030.240119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7030.433858] Lustre: server umount lustre-MDT0000 complete [ 7034.186071] LustreError: 207842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536816 with bad export cookie 17170922803776687188 [ 7034.199540] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7034.528684] Lustre: server umount lustre-MDT0001 complete [ 7043.165788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7043.637194] LustreError: 212036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.661093] LustreError: 212036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 7048.690239] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7057.921231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7062.958902] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7070.249475] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7076.008492] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7082.901806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7088.709831] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7096.735678] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7101.848426] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7102.125763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7107.119348] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7107.476905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7113.188715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7113.198114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7113.201834] LustreError: Skipped 1 previous similar message [ 7113.208780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7113.208795] Lustre: Skipped 7 previous similar messages [ 7113.256739] Lustre: Skipped 4 previous similar messages [ 7117.121680] Lustre: server umount lustre-OST0000 complete [ 7126.613692] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7134.016390] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7164.437762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7171.202107] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7179.123656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7179.152894] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7179.155654] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7179.185157] Lustre: Skipped 1 previous similar message [ 7179.806290] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7182.670425] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 7193.272620] Lustre: server umount lustre-OST0000 complete [ 7203.811758] Lustre: server umount lustre-MDT0000 complete [ 7204.835712] LustreError: 212033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7204.857332] LustreError: 212033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7207.445791] LustreError: 212018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536990 with bad export cookie 17170922803776688112 [ 7207.453908] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7207.457844] LustreError: 212018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7207.760475] Lustre: server umount lustre-MDT0001 complete [ 7217.407754] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:49:58 (1788536998) [ 7224.894297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7225.419283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7225.424781] Lustre: Skipped 6 previous similar messages [ 7230.157556] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7240.081319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7245.332303] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7253.730529] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7259.514670] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7268.323586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7275.538412] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7283.254192] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7289.576075] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7289.885611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7296.056560] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7296.265825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7304.254939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7304.501065] Lustre: lustre-OST0001: new disk, initializing [ 7304.505385] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7304.513622] Lustre: 221377:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7311.328959] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7314.054224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7314.067831] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7314.198578] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7321.886934] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7326.836251] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7327.222377] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7330.766475] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7330.984934] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7334.886229] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7334.899657] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.901154] LustreError: Skipped 2 previous similar messages [ 7334.903328] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7334.903337] Lustre: Skipped 5 previous similar messages [ 7334.953279] Lustre: Skipped 5 previous similar messages [ 7347.170370] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7347.302873] Lustre: server umount lustre-OST0001 complete [ 7354.274742] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7354.524892] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7359.900572] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7360.129400] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7389.648285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7395.312487] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7395.331876] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7395.337928] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7395.357297] Lustre: Skipped 1 previous similar message [ 7396.872353] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7405.395832] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7407.075644] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7415.069360] Lustre: server umount lustre-OST0001 complete [ 7437.854676] Lustre: server umount lustre-OST0000 complete [ 7448.718980] Lustre: server umount lustre-MDT0000 complete [ 7453.055866] LustreError: 217987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788537235 with bad export cookie 17170922803776689176 [ 7453.066644] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7453.077919] LustreError: 217987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7459.461148] Lustre: server umount lustre-MDT0001 complete [ 7483.668743] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:54:25 (1788537265) [ 7490.622695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7490.892189] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7490.933622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7496.475964] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7505.769993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.869282] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7505.914612] Lustre: 226622:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7511.330114] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7519.521528] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7526.099766] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7533.152441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7533.323416] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7539.953156] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7546.837706] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7556.059464] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7556.162988] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7556.298352] LustreError: 228471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7556.312150] LustreError: 228471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 7562.855611] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7570.335168] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7571.901648] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7576.861855] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7577.081355] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7582.076519] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7582.297316] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7589.333598] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff907e8a2c2000.ost_server_uuid 50 [ 7591.054729] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff907e8a2c2000.ost_server_uuid in IDLE state after 0 sec [ 7592.123394] Lustre: Permanently deactivating lustre-OST0001 [ 7592.141065] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7595.719667] Lustre: Permanently reactivating lustre-OST0001 [ 7603.189593] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7603.193245] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7603.194930] LustreError: Skipped 7 previous similar messages [ 7603.201190] Lustre: Skipped 6 previous similar messages [ 7603.203063] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7603.211249] Lustre: Skipped 15 previous similar messages [ 7606.000851] Lustre: server umount lustre-OST0001 complete [ 7624.160203] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7624.339804] Lustre: server umount lustre-OST0000 complete [ 7634.471911] Lustre: server umount lustre-MDT0000 complete [ 7638.189045] LustreError: 226617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788537420 with bad export cookie 17170922803776690646 [ 7638.200249] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7638.201953] LustreError: 226617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7638.618076] Lustre: server umount lustre-MDT0001 complete [ 7665.100795] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:57:26 (1788537446) [ 7672.370550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7672.699457] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7672.731743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7672.740769] Lustre: Skipped 5 previous similar messages [ 7677.784075] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7686.248714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7686.310767] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7686.334171] Lustre: 233319:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7690.111727] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7697.407930] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7702.476663] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7709.514994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7709.828884] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7709.859556] Lustre: Permanently deactivating lustre-OST0000 [ 7709.882979] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7716.118582] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7718.492623] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7723.861452] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7730.872843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7730.976943] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7738.499276] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7741.468329] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7749.299193] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7750.934649] Lustre: Permanently reactivating lustre-OST0000 [ 7750.948207] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7750.952236] Lustre: Skipped 2 previous similar messages [ 7751.390719] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 33 seconds. I think it's dead, and I am evicting it. exp 00000000d09c8be9, cur 1788537534 deadline 1788537531 last 1788537501 [ 7751.392155] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7751.417214] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7751.429787] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7751.468224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7779.296461] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7779.353455] Lustre: server umount lustre-OST0001 complete [ 7797.216715] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7797.483402] Lustre: server umount lustre-OST0000 complete [ 7807.757746] Lustre: server umount lustre-MDT0000 complete [ 7812.283873] LustreError: 233312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788537595 with bad export cookie 17170922803776692480 [ 7812.296858] LustreError: 233312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7812.303367] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7812.813591] Lustre: server umount lustre-MDT0001 complete [ 7824.774360] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 7827.878707] Key type lgssc unregistered [ 7828.077540] LNet: 238201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7828.095265] LNetError: 238201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7829.171777] LNet: Removed LNI 192.168.202.126@tcp [ 7829.878068] Key type .llcrypt unregistered [ 7829.883970] Key type ._llcrypt unregistered [ 7859.564529] Key type ._llcrypt registered [ 7859.569137] Key type .llcrypt registered [ 7859.697504] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 7860.804147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7860.849126] alg: No test for adler32 (adler32-zlib) [ 7862.102609] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 7862.418106] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 7864.048479] Key type lgssc registered [ 7864.961285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7876.133250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7876.159532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7877.630837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7883.350305] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7887.849282] LustreError: 239665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7887.867655] LustreError: 239665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7892.968180] LustreError: 239664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7892.982924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7893.092610] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7893.119778] Lustre: 239652:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7893.128810] Lustre: Permanently deactivating lustre-MDT0001 [ 7893.140647] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7893.480936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7896.732580] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7896.740552] LustreError: 239695:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7896.750737] LustreError: 239695:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7896.763244] LustreError: 239695:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 1, failed: rc = -108 [ 7898.554302] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7906.755517] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7913.076720] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7920.378852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7920.942268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7922.997198] LustreError: 241518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7923.018251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7927.082476] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7928.294922] LustreError: 241520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.413412] LustreError: 241518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7935.176429] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7938.557055] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7943.356422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7943.564973] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7948.814325] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7948.821505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7951.062526] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7959.537821] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7963.533720] Lustre: Permanently reactivating lustre-MDT0001 [ 7963.558636] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7963.573760] Lustre: Skipped 2 previous similar messages [ 7968.578084] Lustre: lustre-MDT0001: Client 4c473b29-8f68-48c0-acb6-3a98703347b7 (at 192.168.202.26@tcp) reconnecting [ 7976.048444] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7977.941576] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7984.445173] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7984.702652] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7990.662337] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7990.922594] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7994.146372] Lustre: Permanently deactivating lustre-MDT0001 [ 7994.166621] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7994.170485] Lustre: Skipped 2 previous similar messages [ 8003.178560] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8015.345250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8015.364539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8015.376178] Lustre: Skipped 2 previous similar messages [ 8018.619231] Lustre: server umount lustre-MDT0000 complete [ 8020.464666] LustreError: 239665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8020.480095] LustreError: 239665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8022.996233] LustreError: 239645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788537805 with bad export cookie 738748599275329852 [ 8023.015272] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8023.015758] LustreError: 239645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8023.265717] Lustre: server umount lustre-MDT0001 complete [ 8029.565372] Lustre: server umount lustre-OST0000 complete [ 8043.611339] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:03:44 (1788537824) [ 8054.573450] Lustre: server umount lustre-OST0001 complete [ 8061.942323] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 8072.868173] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 8116.235862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8116.465179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8116.475419] Lustre: Skipped 2 previous similar messages [ 8116.500874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8116.647868] Lustre: lustre-MDT0000: new disk, initializing [ 8116.784705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8116.814863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8122.145556] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8133.755246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8133.855564] Lustre: 247959:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8133.939852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8133.943862] Lustre: Skipped 1 previous similar message [ 8133.997653] Lustre: lustre-MDT0001: new disk, initializing [ 8134.081745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8134.122034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8134.133195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8138.579722] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8148.778206] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8155.130500] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8162.602698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8163.092068] Lustre: lustre-OST0000: new disk, initializing [ 8163.096822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8163.103300] Lustre: 249927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8163.229321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8168.582037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8168.604137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8168.744854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8170.383885] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8181.591931] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8183.872150] LustreError: 247965:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8193.282344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8193.457275] Lustre: lustre-OST0001: new disk, initializing [ 8193.459863] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8193.463707] Lustre: 251041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8193.944112] LustreError: 247965:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8193.988841] LustreError: 250694:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8200.442332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8200.464620] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8200.587553] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8200.610642] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8204.048659] LustreError: 250694:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8210.732728] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8212.347712] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8216.047639] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8216.053800] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8216.058724] Lustre: Skipped 1 previous similar message [ 8216.079060] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8216.084066] Lustre: Skipped 4 previous similar messages [ 8221.184904] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8221.189550] Lustre: Skipped 1 previous similar message [ 8226.283250] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8226.308325] Lustre: Skipped 1 previous similar message [ 8229.344241] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8229.485312] Lustre: server umount lustre-OST0001 complete [ 8231.399247] LustreError: 250838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8231.421418] LustreError: 250838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8250.851565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8250.860364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8250.862252] LustreError: Skipped 2 previous similar messages [ 8250.870201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8250.875036] Lustre: Skipped 2 previous similar messages [ 8250.893074] Lustre: Skipped 1 previous similar message [ 8253.190485] Lustre: server umount lustre-OST0000 complete [ 8262.118997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8262.121611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8262.121966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8262.132243] Lustre: Skipped 1 previous similar message [ 8262.162951] Lustre: Skipped 3 previous similar messages [ 8263.884880] Lustre: server umount lustre-MDT0000 complete [ 8267.241989] LustreError: 247969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8267.266928] LustreError: 247969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 8267.826136] LustreError: 247950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788538050 with bad export cookie 738748599275332848 [ 8267.832930] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8267.837505] LustreError: 247950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8274.343325] Lustre: server umount lustre-MDT0001 complete [ 8284.701812] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 8287.540236] Key type lgssc unregistered [ 8287.802213] LNet: 253076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8287.814391] LNetError: 253076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8288.870863] LNet: Removed LNI 192.168.202.126@tcp [ 8289.889067] Key type .llcrypt unregistered [ 8289.891434] Key type ._llcrypt unregistered [ 8334.945665] Key type ._llcrypt registered [ 8334.948277] Key type .llcrypt registered [ 8335.079214] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 8337.018654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8337.205844] alg: No test for adler32 (adler32-zlib) [ 8338.402417] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 8338.860790] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 8340.681204] Key type lgssc registered [ 8342.738717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8352.708775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8352.721696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8353.894307] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8353.902441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8354.004858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8358.728246] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8368.341814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8368.425235] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8368.446460] Lustre: 255284:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8368.849178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8374.442424] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8383.662380] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8390.708490] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8398.724426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8398.967995] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8399.290792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8406.138766] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8406.778416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8414.848232] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8468.453144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8468.456405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8468.470328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8468.477708] Lustre: Skipped 1 previous similar message [ 8468.512779] LustreError: Skipped 1 previous similar message [ 8472.453162] Lustre: server umount lustre-OST0000 complete [ 8478.240497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8492.026166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8492.475766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8499.162959] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8507.635551] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8508.860245] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8508.877099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8508.890541] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8549.866903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8549.870614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8549.876919] LustreError: Skipped 1 previous similar message [ 8549.880961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8549.890916] Lustre: Skipped 1 previous similar message [ 8549.903352] Lustre: Skipped 1 previous similar message [ 8554.284359] Lustre: server umount lustre-OST0000 complete [ 8558.561539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8558.576111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8558.589271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8558.598963] Lustre: Skipped 1 previous similar message [ 8564.731661] Lustre: server umount lustre-MDT0000 complete [ 8569.045455] LustreError: 255276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788538351 with bad export cookie 12599068745113628857 [ 8569.050347] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8569.056242] LustreError: 255276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8569.532199] Lustre: server umount lustre-MDT0001 complete [ 8581.199914] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 8584.114561] Key type lgssc unregistered [ 8584.419730] LNet: 260710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8584.438071] LNetError: 260710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8584.459558] LNet: Removed LNI 192.168.202.126@tcp [ 8585.548611] Key type .llcrypt unregistered [ 8585.551410] Key type ._llcrypt unregistered [ 8611.956356] Key type ._llcrypt registered [ 8611.957403] Key type .llcrypt registered [ 8612.035850] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 8613.384498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8613.409952] alg: No test for adler32 (adler32-zlib) [ 8614.385995] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 8614.633177] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 8616.265310] Key type lgssc registered [ 8617.118276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8625.984805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8626.013787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8627.653781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8633.142289] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8637.928035] LustreError: 262124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8637.952886] LustreError: 262124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8643.045805] LustreError: 262123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8645.483101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8646.056429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8652.784951] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8662.488392] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8668.316263] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8675.421745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8675.772683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8681.747429] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8689.020387] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8697.278636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8727.010364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8727.016495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8727.022942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8728.047806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8728.049561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8728.051849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8728.078346] Lustre: Skipped 1 previous similar message [ 8730.602340] Lustre: server umount lustre-OST0000 complete [ 8738.272992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8738.275871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8738.277133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8738.278404] Lustre: Skipped 1 previous similar message [ 8740.947591] Lustre: server umount lustre-MDT0000 complete [ 8743.395649] LustreError: 262123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8743.414581] LustreError: 262123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8744.721989] LustreError: 262104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788538527 with bad export cookie 9677419722188171370 [ 8744.732170] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8744.738076] LustreError: 262104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8745.143935] Lustre: server umount lustre-MDT0001 complete [ 8756.166895] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 8758.898715] Key type lgssc unregistered [ 8759.316051] LNet: 266452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8759.325550] LNetError: 266452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8759.352626] LNet: Removed LNI 192.168.202.126@tcp [ 8760.584268] Key type .llcrypt unregistered [ 8760.588974] Key type ._llcrypt unregistered [ 8777.305919] Key type ._llcrypt registered [ 8777.310495] Key type .llcrypt registered [ 8777.434057] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 8778.481719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8778.607932] alg: No test for adler32 (adler32-zlib) [ 8779.808237] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 8780.205898] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 8781.888178] Key type lgssc registered [ 8783.030921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8790.821293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8790.845516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8792.359957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8797.168526] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8802.786896] LustreError: 267669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8802.810989] LustreError: 267669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8804.786671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8805.133538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8810.558162] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8823.020868] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8830.357301] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8839.503561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8839.976617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8848.153899] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8856.725756] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8862.524277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8878.049968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8878.063710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8878.069360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8878.088266] Lustre: Skipped 1 previous similar message [ 8878.101238] Lustre: Skipped 1 previous similar message [ 8879.819525] Lustre: server umount lustre-OST0000 complete [ 8887.266316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8887.285884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8887.310811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8888.299266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8888.323545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8888.338826] Lustre: Skipped 1 previous similar message [ 8890.572030] Lustre: server umount lustre-MDT0000 complete [ 8893.410491] LustreError: 268643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8893.461656] LustreError: 268643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8895.561967] LustreError: 267649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788538678 with bad export cookie 15111475915526220214 [ 8895.564524] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8895.573133] LustreError: 267649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8896.463763] Lustre: server umount lustre-MDT0001 complete [ 8908.495442] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 8911.242980] Key type lgssc unregistered [ 8911.545459] LNet: 271373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8911.552907] LNetError: 271373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8911.571217] LNet: Removed LNI 192.168.202.126@tcp [ 8912.404437] Key type .llcrypt unregistered [ 8912.410371] Key type ._llcrypt unregistered [ 8940.871555] Key type ._llcrypt registered [ 8940.874755] Key type .llcrypt registered [ 8940.983623] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 8942.466564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8942.485333] alg: No test for adler32 (adler32-zlib) [ 8943.508469] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 8943.817680] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 8945.488142] Key type lgssc registered [ 8946.333821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8953.249413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8953.263543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8954.608530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8959.708415] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8965.099628] LustreError: 272786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8965.117420] LustreError: 272786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8969.549965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8969.947045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8975.377570] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8984.959667] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8991.170727] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8997.986529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8998.329733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8999.361848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9005.274626] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9012.153861] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9040.357051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9040.370498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9040.371444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9040.381318] Lustre: Skipped 1 previous similar message [ 9042.409760] Lustre: server umount lustre-OST0000 complete [ 9047.009887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9047.027800] LustreError: Skipped 1 previous similar message [ 9047.032981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9047.053025] Lustre: Skipped 1 previous similar message [ 9047.068387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9050.594857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9050.597053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9055.714701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9055.724153] Lustre: Skipped 2 previous similar messages [ 9060.320771] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9060.682276] Lustre: server umount lustre-MDT0000 complete [ 9060.841318] LustreError: 272781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9060.862970] LustreError: 272781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9065.065772] LustreError: 272766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788538847 with bad export cookie 16801278424522505928 [ 9065.074125] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9065.082905] LustreError: 272766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9065.735851] Lustre: server umount lustre-MDT0001 complete [ 9077.737448] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 9080.291420] Key type lgssc unregistered [ 9080.601745] LNet: 277095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9080.605524] LNetError: 277095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9080.624295] LNet: Removed LNI 192.168.202.126@tcp [ 9081.418929] Key type .llcrypt unregistered [ 9081.421525] Key type ._llcrypt unregistered [ 9105.003273] Key type ._llcrypt registered [ 9105.008371] Key type .llcrypt registered [ 9105.242837] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 9107.434144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9107.495839] alg: No test for adler32 (adler32-zlib) [ 9108.608257] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 9108.888763] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 9110.794434] Key type lgssc registered [ 9112.606338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9121.504840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9121.521325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9122.958705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9126.520732] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9132.526631] LustreError: 278330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9132.544495] LustreError: 278330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9136.401779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9137.284588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9142.699169] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9150.159155] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9155.073893] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9162.603661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9163.021577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9168.179139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9170.239867] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9176.695054] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9193.954111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9193.959046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.968525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9197.025602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9197.026863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9198.590568] Lustre: server umount lustre-OST0000 complete [ 9203.683658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9203.692273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9203.716598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9203.725287] Lustre: Skipped 1 previous similar message [ 9207.267329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9207.268564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9208.659408] Lustre: server umount lustre-MDT0000 complete [ 9212.268493] LustreError: 278295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788538995 with bad export cookie 2442537143015309259 [ 9212.282087] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9212.794980] Lustre: server umount lustre-MDT0001 complete [ 9225.003875] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 9227.665779] Key type lgssc unregistered [ 9227.962972] LNet: 282031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9227.969232] LNetError: 282031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9227.979665] LNet: Removed LNI 192.168.202.126@tcp [ 9228.902958] Key type .llcrypt unregistered [ 9228.907691] Key type ._llcrypt unregistered [ 9239.397928] Key type ._llcrypt registered [ 9239.402267] Key type .llcrypt registered [ 9239.594568] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9257.149899] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 9271.867912] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 9272.946335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9272.977362] alg: No test for adler32 (adler32-zlib) [ 9274.049721] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 9274.323451] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 9276.009653] Key type lgssc registered [ 9277.047845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9318.135596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9318.184748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9319.556034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9319.603269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9319.713233] Lustre: lustre-MDT0000: new disk, initializing [ 9319.805851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9319.846155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9325.342978] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9336.070687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9336.193303] Lustre: 285581:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9336.307875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9336.311111] Lustre: Skipped 1 previous similar message [ 9336.368947] Lustre: lustre-MDT0001: new disk, initializing [ 9336.493820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9336.557211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9336.568702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9341.438616] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9351.448995] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9357.893247] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9364.093069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9364.366985] Lustre: lustre-OST0000: new disk, initializing [ 9364.386856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9364.394766] Lustre: 287547:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9364.522624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9370.868692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9370.888956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9371.185417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9373.475720] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9386.507122] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9391.913393] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9392.084880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9397.269315] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9397.539690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9401.826808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9401.830568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9401.835538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9401.850648] Lustre: Skipped 1 previous similar message [ 9401.866527] Lustre: Skipped 1 previous similar message [ 9406.945432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9406.953189] Lustre: Skipped 1 previous similar message [ 9412.069270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9412.081351] Lustre: Skipped 1 previous similar message [ 9414.112421] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9414.567794] Lustre: server umount lustre-OST0000 complete [ 9422.305206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9422.310987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9422.329903] Lustre: Skipped 1 previous similar message [ 9422.350024] Lustre: Skipped 1 previous similar message [ 9425.373496] Lustre: server umount lustre-MDT0000 complete [ 9429.962887] LustreError: 285571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788539212 with bad export cookie 12523184671440578580 [ 9429.967364] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9429.968542] LustreError: 285571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9430.279800] Lustre: server umount lustre-MDT0001 complete [ 9439.640857] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:27:00 (1788539220) [ 9447.642542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9447.986686] LustreError: 289963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9448.000778] LustreError: 289963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9448.067280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9453.131606] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9453.549059] LustreError: 289963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9453.565802] LustreError: 289963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9457.644605] LustreError: 289964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9462.766140] LustreError: 289963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9463.701824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9464.288254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9469.729984] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9479.266917] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9486.055720] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9495.126414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9495.538222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9503.854938] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9513.367725] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9528.290798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9528.300687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9528.301079] LustreError: Skipped 1 previous similar message [ 9528.309946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9528.329643] Lustre: Skipped 1 previous similar message [ 9534.484698] Lustre: server umount lustre-OST0000 complete [ 9541.089616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9541.104996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9541.144540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9541.149840] Lustre: Skipped 3 previous similar messages [ 9545.289148] Lustre: server umount lustre-MDT0000 complete [ 9548.781646] LustreError: 289963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9548.802291] LustreError: 289963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9549.192480] LustreError: 289944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788539331 with bad export cookie 12523184671440579504 [ 9549.201809] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9549.203652] LustreError: 289944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9549.657806] Lustre: server umount lustre-MDT0001 complete [ 9562.332278] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 9565.440873] Key type lgssc unregistered [ 9565.903495] LNet: 293523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9565.909502] LNetError: 293523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9565.938531] LNet: Removed LNI 192.168.202.126@tcp [ 9566.900450] Key type .llcrypt unregistered [ 9566.903508] Key type ._llcrypt unregistered [ 9597.767831] Key type ._llcrypt registered [ 9597.769493] Key type .llcrypt registered [ 9597.894878] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 9617.268424] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 9618.877872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9618.910104] alg: No test for adler32 (adler32-zlib) [ 9620.052444] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [ 9620.351258] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 9622.080222] Key type lgssc registered [ 9623.683882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9678.077172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9678.097792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9679.419886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9679.468572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9679.576098] Lustre: lustre-MDT0000: new disk, initializing [ 9679.678384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9679.716634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9684.779984] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9696.427721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9696.554672] Lustre: 297352:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9696.628305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9696.631248] Lustre: Skipped 1 previous similar message [ 9696.692659] Lustre: lustre-MDT0001: new disk, initializing [ 9696.775032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9696.827536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9696.840228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9701.588301] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9712.610690] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9718.743467] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9726.044341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9726.272526] Lustre: lustre-OST2710: new disk, initializing [ 9726.275714] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9726.280894] Lustre: 299320:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9726.363025] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9733.309491] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9734.273061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9734.292524] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9734.452643] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9744.649981] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9752.294836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9752.515391] Lustre: lustre-OST03e8: new disk, initializing [ 9752.520875] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9752.526640] Lustre: 300387:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9752.658317] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9759.782888] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9761.560785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9761.576730] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9761.687276] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9769.133746] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9777.484334] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9777.692646] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9783.461379] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9783.728513] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9789.148826] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9789.386858] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9794.108850] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9794.373492] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9800.279541] Lustre: server umount lustre-MDT0000 complete [ 9802.726303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9802.741306] Lustre: Skipped 3 previous similar messages [ 9804.388889] LustreError: 297342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788539587 with bad export cookie 8754605479103469611 [ 9804.399311] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9804.421061] LustreError: 297342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9805.103100] Lustre: server umount lustre-MDT0001 complete [ 9819.752134] Lustre: server umount lustre-OST2710 complete [ 9834.345516] Lustre: server umount lustre-OST03e8 complete [ 9859.110142] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 9873.145541] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 9911.323569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9911.574495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9911.599991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9911.668906] Lustre: lustre-MDT0000: new disk, initializing [ 9911.744987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9911.761508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9916.637636] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9928.106637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9928.227173] Lustre: 305638:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9928.317469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9928.322650] Lustre: Skipped 1 previous similar message [ 9928.383618] Lustre: lustre-MDT0001: new disk, initializing [ 9928.465143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9928.493631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9928.509916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9933.723530] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9944.333746] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9950.962831] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9959.232774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9959.599528] Lustre: lustre-OST0000: new disk, initializing [ 9959.614044] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9959.620590] Lustre: 307604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9959.773474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9965.289150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9965.303746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9965.477843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9967.791322] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9977.904958] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9983.528584] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9983.921326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9989.386660] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9989.612381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9996.262866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9996.264436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9996.278169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9996.303568] Lustre: Skipped 1 previous similar message [ 9998.076553] Lustre: server umount lustre-OST0000 complete [10005.482825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10005.495507] LustreError: Skipped 1 previous similar message [10005.508380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10005.554376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10005.566107] Lustre: Skipped 1 previous similar message [10011.617799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10011.622155] Lustre: Skipped 3 previous similar messages [10016.224913] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10016.532964] Lustre: server umount lustre-MDT0000 complete [10016.739342] LustreError: 305648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10016.748041] LustreError: 305648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10020.634486] LustreError: 305630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788539803 with bad export cookie 8754605479103471403 [10020.638912] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10020.644443] LustreError: 305630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10021.065790] Lustre: server umount lustre-MDT0001 complete [10030.844048] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:36:51 (1788539811) [10032.484245] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10034.224714] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:36:55 (1788539815) [10037.880573] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [10069.267077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10069.570205] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10069.607219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10069.722233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10075.895883] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10086.614269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10086.730959] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10086.765960] Lustre: 311468:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10091.732968] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10112.966859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10113.258169] Lustre: Denying initial registration attempt from nid 192.168.202.126@tcp, specified as failover [10113.273218] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10113.285895] LustreError: 312920:0:(tgt_mount.c:2742:server_fill_super()) Unable to start targets: -99 [10113.303290] LustreError: 312920:0:(tgt_mount.c:2192:server_put_super()) no obd lustre-OST0000 [10113.317185] LustreError: 312920:0:(tgt_mount.c:130:server_deregister_mount()) lustre-OST0000 not registered [10113.399531] Lustre: server umount lustre-OST0000 complete [10113.403745] LustreError: 312920:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10122.721556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10122.737353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10122.776839] Lustre: Skipped 1 previous similar message [10122.795301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10126.414985] Lustre: server umount lustre-MDT0000 complete [10129.383630] LustreError: 311480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.418102] LustreError: 311480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10130.616040] LustreError: 311460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788539913 with bad export cookie 8754605479103472327 [10130.643598] LustreError: 311460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10130.652991] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10131.131110] Lustre: server umount lustre-MDT0001 complete [10143.470589] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [10146.268550] Key type lgssc unregistered [10146.715268] LNet: 314041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10146.730633] LNetError: 314041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10146.757294] LNet: Removed LNI 192.168.202.126@tcp [10147.691835] Key type .llcrypt unregistered [10147.693685] Key type ._llcrypt unregistered [10165.040384] Key type ._llcrypt registered [10165.042030] Key type .llcrypt registered [10165.124833] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [10166.343192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10166.369525] alg: No test for adler32 (adler32-zlib) [10167.457085] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [10167.884268] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [10169.664201] Key type lgssc registered [10170.974800] Lustre: Echo OBD driver; http://www.lustre.org/ [10216.174820] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [10225.066826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10225.090761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10226.452331] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10226.468878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10226.598173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10231.240912] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10239.384837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10239.437882] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10239.451814] Lustre: 316783:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10239.669852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10244.171638] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10251.578527] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10258.390839] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10279.911087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10280.060295] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10280.273360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10286.629853] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10295.088834] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10299.364388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10299.368175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10299.382513] LustreError: Skipped 1 previous similar message [10299.386722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10299.419026] Lustre: Skipped 1 previous similar message [10304.484789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10304.508861] Lustre: Skipped 1 previous similar message [10309.604419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10309.613614] Lustre: Skipped 1 previous similar message [10312.160143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10312.374604] Lustre: server umount lustre-OST0000 complete [10316.772213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10316.778590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10316.788655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10316.794217] Lustre: Skipped 1 previous similar message [10319.863515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10322.350914] Lustre: server umount lustre-MDT0000 complete [10326.380312] LustreError: 316775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788540109 with bad export cookie 9948029486511336912 [10326.385159] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10326.401560] LustreError: 316775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10326.705816] Lustre: server umount lustre-MDT0001 complete [10340.633938] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [10344.506499] Key type lgssc unregistered [10344.835418] LNet: 320358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10344.845725] LNetError: 320358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10344.871719] LNet: Removed LNI 192.168.202.126@tcp [10345.883690] Key type .llcrypt unregistered [10345.887538] Key type ._llcrypt unregistered [10369.479578] Key type ._llcrypt registered [10369.487753] Key type .llcrypt registered [10369.568417] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [10370.399813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10370.446079] alg: No test for adler32 (adler32-zlib) [10371.448776] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [10371.682069] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [10373.336808] Key type lgssc registered [10374.304689] Lustre: Echo OBD driver; http://www.lustre.org/ [10380.697910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10380.718257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10382.323585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10386.634681] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10392.546408] LustreError: 321770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10392.580813] LustreError: 321770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10395.799550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10396.118710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10401.735572] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10410.092110] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10416.200416] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10425.602405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10426.004511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10432.970100] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10439.917716] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10450.920287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10450.931547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10450.939040] Lustre: Skipped 2 previous similar messages [10450.969825] Lustre: Skipped 2 previous similar messages [10451.774214] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.26@tcp (stopping) [10454.594521] Lustre: server umount lustre-MDT0000 complete [10456.034634] LustreError: 322500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10456.047771] LustreError: 322500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10458.415447] LustreError: 321752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788540241 with bad export cookie 649979818570318822 [10458.422022] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10458.430837] LustreError: 321752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10461.153863] LustreError: 321766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10461.157015] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10461.168113] LustreError: 321766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10461.181161] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10463.201416] LustreError: 321767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10463.204457] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10463.218563] LustreError: 321767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10463.233450] Lustre: Skipped 1 previous similar message [10464.851696] Lustre: server umount lustre-MDT0001 complete [10471.280294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10480.257605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10483.682435] LustreError: 321080:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000025db48c0 x1875420280554880/t0(0) o250->MGC192.168.202.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10484.041400] LustreError: 323606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10484.104382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10484.160646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10488.814219] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10493.356331] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10493.370699] LustreError: 325365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10493.385518] LustreError: 325365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10498.519539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10499.180436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10504.168824] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10504.411510] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10513.527299] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10515.242624] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10521.585610] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10523.107381] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10529.768580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10529.781674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10529.782456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10529.809364] Lustre: Skipped 1 previous similar message [10534.866918] Lustre: server umount lustre-OST0000 complete [10539.381391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10539.384153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10539.395231] Lustre: Skipped 2 previous similar messages [10539.411496] Lustre: Skipped 1 previous similar message [10545.800616] Lustre: server umount lustre-MDT0000 complete [10549.731771] LustreError: 325369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10549.761229] LustreError: 325369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10549.851865] LustreError: 321750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788540332 with bad export cookie 649979818570326025 [10549.859854] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10549.868358] LustreError: 321750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10550.407564] Lustre: server umount lustre-MDT0001 complete [10563.357597] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [10566.430861] Key type lgssc unregistered [10566.822650] LNet: 328038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10566.840480] LNetError: 328038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10567.849245] LNet: Removed LNI 192.168.202.126@tcp [10568.770979] Key type .llcrypt unregistered [10568.773430] Key type ._llcrypt unregistered [10594.344690] Key type ._llcrypt registered [10594.348301] Key type .llcrypt registered [10594.490039] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [10595.641463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10595.701345] alg: No test for adler32 (adler32-zlib) [10597.098124] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [10597.385295] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [10599.072202] Key type lgssc registered [10600.820736] Lustre: Echo OBD driver; http://www.lustre.org/ [10610.057424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10610.083837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10611.731495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10616.605702] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10621.926638] LustreError: 329454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10621.967989] LustreError: 329454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10626.764348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10627.045910] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10627.190987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10631.915900] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10642.434925] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10650.776180] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10654.192490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10654.203068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10654.210035] Lustre: Skipped 1 previous similar message [10654.230120] Lustre: Skipped 1 previous similar message [10659.297832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10659.307568] Lustre: Skipped 1 previous similar message [10660.659634] Lustre: server umount lustre-MDT0000 complete [10664.420087] LustreError: 330590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10664.446466] LustreError: 330590:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10664.692140] LustreError: 329435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788540447 with bad export cookie 5109805800639530861 [10664.704393] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10664.707287] LustreError: 329435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10665.140971] Lustre: server umount lustre-MDT0001 complete [10680.902646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10681.214281] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10681.271195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10681.531313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10686.742901] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10696.260781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10696.434299] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10696.503325] Lustre: 332043:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10696.951813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10702.854142] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10712.222739] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10719.764034] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10731.828298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10732.182465] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10732.193939] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10732.209352] LustreError: 333930:0:(tgt_mount.c:2742:server_fill_super()) Unable to start targets: -2 [10732.218601] LustreError: 333930:0:(tgt_mount.c:2192:server_put_super()) no obd lustre-OST0000 [10732.229124] LustreError: 333930:0:(tgt_mount.c:130:server_deregister_mount()) lustre-OST0000 not registered [10732.343812] Lustre: server umount lustre-OST0000 complete [10732.346169] LustreError: 333930:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10743.743682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10744.060537] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10744.300877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10751.439230] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10753.816537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10760.797227] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10769.154421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10769.219896] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10769.372402] Lustre: lustre-OST0001: new disk, initializing [10769.375542] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10769.390950] Lustre: 335472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10769.497068] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10775.933362] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10777.408470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10777.421788] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10777.630235] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10783.369710] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10787.818180] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10787.821138] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10787.859298] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10787.878982] LustreError: Skipped 1 previous similar message [10792.320088] Lustre: server umount lustre-OST0001 complete [10792.940474] LustreError: 334398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10792.965119] LustreError: 334398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10798.051747] LustreError: 334397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10798.059894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10798.068322] LustreError: 334397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10798.104160] Lustre: Skipped 1 previous similar message [10798.117843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10798.120194] Lustre: Skipped 2 previous similar messages [10802.483613] Lustre: server umount lustre-OST0000 complete [10808.301839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10808.311225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10808.327668] Lustre: Skipped 2 previous similar messages [10808.340939] Lustre: Skipped 1 previous similar message [10812.905636] Lustre: server umount lustre-MDT0000 complete [10813.417140] LustreError: 332050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10813.439213] LustreError: 332050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10816.873072] LustreError: 332036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788540599 with bad export cookie 5109805800639531428 [10816.877922] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10816.884576] LustreError: 332036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10817.256212] Lustre: server umount lustre-MDT0001 complete [10829.228924] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [10831.881449] Key type lgssc unregistered [10832.187961] LNet: 337430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10832.193452] LNetError: 337430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10832.210646] LNet: Removed LNI 192.168.202.126@tcp [10833.243832] Key type .llcrypt unregistered [10833.245576] Key type ._llcrypt unregistered [10911.075681] Key type ._llcrypt registered [10911.078409] Key type .llcrypt registered [10911.207352] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [10927.050147] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [10928.262870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10928.293216] alg: No test for adler32 (adler32-zlib) [10929.270247] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [10929.439203] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [10931.064156] Key type lgssc registered [10931.786569] Lustre: Echo OBD driver; http://www.lustre.org/ [10980.040716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10980.078574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10981.342142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10981.387528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10981.505544] Lustre: lustre-MDT0000: new disk, initializing [10981.660054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10981.680929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10987.770586] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11000.214667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11000.435199] Lustre: 342363:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11000.636715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11000.645375] Lustre: Skipped 1 previous similar message [11000.893551] Lustre: lustre-MDT0001: new disk, initializing [11001.108988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11001.218874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11001.250371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11006.401592] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11017.695095] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11025.521956] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11033.817673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11034.240400] Lustre: lustre-OST0000: new disk, initializing [11034.249700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11034.263679] Lustre: 344334:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11034.384038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11042.748180] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11042.977022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11042.985847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11043.170698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11052.940440] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11058.237473] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11058.457451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11063.492871] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11063.732458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11068.899537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11068.906571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11068.921772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11072.084662] Lustre: server umount lustre-OST0000 complete [11078.113741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11078.130463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11078.152239] Lustre: Skipped 1 previous similar message [11078.161087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11078.163784] Lustre: Skipped 1 previous similar message [11079.140726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11083.115559] Lustre: server umount lustre-MDT0000 complete [11088.269107] LustreError: 342355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788540871 with bad export cookie 17764028229967313076 [11088.282034] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11088.286526] LustreError: 342355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11088.518396] Lustre: server umount lustre-MDT0001 complete [11098.284944] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:54:39 (1788540879) [11107.421337] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:54:48 (1788540888) [11116.880542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11117.420056] LustreError: 347091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11117.442394] LustreError: 347091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11117.497572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11122.623439] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11122.660210] LustreError: 347092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11127.787590] LustreError: 347091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11132.610075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11132.897583] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11132.908318] Lustre: Skipped 2 previous similar messages [11133.056906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11138.740542] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11146.493621] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11152.217512] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11159.544530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11159.809147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11165.597396] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11176.980827] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11186.486220] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11189.150217] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11191.633894] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11194.046783] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11196.840397] Lustre: Failing over lustre-MDT0000 [11197.121213] Lustre: server umount lustre-MDT0000 complete [11197.923687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11197.933819] LustreError: 347091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11197.940438] Lustre: Skipped 1 previous similar message [11197.959537] LustreError: 347091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11203.043953] LustreError: 347092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11203.058518] LustreError: 347092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11203.064377] LustreError: MGC192.168.202.126@tcp: operation ldlm_enqueue to node 0@lo failed: rc = -107 [11203.064644] LustreError: 347073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788540985 with bad export cookie 17764028229967314000 [11203.066716] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11203.067873] Lustre: 349981:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [11203.093259] Lustre: 347083:0:(mgc_request.c:2298:mgc_process_log()) MGC192.168.202.126@tcp: IR log lustre-mdtir failed, not fatal: rc = -107 [11208.168082] Lustre: 349983:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [11209.187255] Lustre: 349981:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [11214.310488] Lustre: 349983:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [11214.312281] LustreError: 348237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11214.338062] LustreError: 348237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11216.343080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11217.007473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11217.074193] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11220.262078] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11221.712617] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11221.996481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11222.011825] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11222.057821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11232.094931] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11234.343919] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11240.096955] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11242.944463] Lustre: Failing over lustre-MDT0000 [11243.816603] Lustre: server umount lustre-MDT0000 complete [11245.900677] LustreError: 347088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11245.942795] LustreError: 347088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11247.591295] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11247.626249] Lustre: Skipped 2 previous similar messages [11247.644279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11251.190801] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11251.217980] Lustre: 351183:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [11251.220723] Lustre: 347083:0:(mgc_request.c:2298:mgc_process_log()) MGC192.168.202.126@tcp: IR log lustre-mdtir failed, not fatal: rc = -107 [11259.887207] Lustre: 351183:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [11259.907332] Lustre: 351183:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 2 previous similar messages [11265.895407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11266.395749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11266.454554] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11269.549540] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11271.691415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11271.703818] Lustre: Skipped 2 previous similar messages [11271.740381] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11271.815503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11272.340487] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11282.216868] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11284.145446] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11292.137811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11292.147296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11292.154280] LustreError: Skipped 1 previous similar message [11292.157985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11292.175664] Lustre: Skipped 2 previous similar messages [11292.204625] Lustre: Skipped 2 previous similar messages [11297.253179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11297.269628] Lustre: Skipped 2 previous similar messages [11297.858378] Lustre: server umount lustre-MDT0000 complete [11302.326717] LustreError: 347071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541085 with bad export cookie 17764028229967316072 [11302.341143] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11302.367804] LustreError: 347071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11302.369780] LustreError: 347087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11302.379158] LustreError: 347087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11302.991139] Lustre: server umount lustre-MDT0001 complete [11309.839511] Lustre: server umount lustre-OST0000 complete [11328.065385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11334.167703] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11343.831663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11344.305234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11344.314400] Lustre: Skipped 1 previous similar message [11349.806432] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11357.862725] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11364.171958] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11372.232672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11374.649949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11380.253435] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11390.219242] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11399.152014] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11399.154946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11399.159615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11399.159624] Lustre: Skipped 1 previous similar message [11399.165485] Lustre: Skipped 3 previous similar messages [11410.400268] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11410.576736] Lustre: server umount lustre-OST0000 complete [11419.619492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11419.644079] Lustre: Skipped 6 previous similar messages [11423.021914] Lustre: server umount lustre-MDT0000 complete [11424.737694] LustreError: 353440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11424.781807] LustreError: 353440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11428.359365] LustreError: 353425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541211 with bad export cookie 17764028229967316884 [11428.366516] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11428.374381] LustreError: 353425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11429.109211] Lustre: server umount lustre-MDT0001 complete [11443.855657] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [11446.598469] Key type lgssc unregistered [11446.966200] LNet: 356979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11446.979723] LNetError: 356979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11448.039872] LNet: Removed LNI 192.168.202.126@tcp [11449.158252] Key type .llcrypt unregistered [11449.164167] Key type ._llcrypt unregistered [11477.779512] Key type ._llcrypt registered [11477.781831] Key type .llcrypt registered [11477.893781] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [11478.918750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11479.034039] alg: No test for adler32 (adler32-zlib) [11480.479829] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [11480.824747] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [11482.601871] Key type lgssc registered [11484.457465] Lustre: Echo OBD driver; http://www.lustre.org/ [11493.240917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11493.260071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11494.727874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11499.313428] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11505.124283] LustreError: 358506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11505.145745] LustreError: 358506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11507.888961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11508.352300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11513.929448] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11522.079217] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11528.437687] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11536.174284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11536.652921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11539.774460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11544.033369] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11551.975716] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11564.513235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11564.532085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11565.854246] Lustre: server umount lustre-MDT0000 complete [11569.406745] LustreError: 361085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541352 with bad export cookie 9418515946312004593 [11569.407406] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11569.417330] LustreError: 361085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11569.633707] LustreError: 358503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11569.635710] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11569.655826] Lustre: Skipped 2 previous similar messages [11569.726917] Lustre: server umount lustre-MDT0001 complete [11575.995793] Lustre: server umount lustre-OST0000 complete [11591.225849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11591.765174] LustreError: 362216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11591.784824] LustreError: 362216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11591.882324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11596.790813] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11597.288789] LustreError: 362217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11601.380511] LustreError: 362216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11605.467750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11605.783394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11610.374403] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11618.666259] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11624.807897] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11632.341789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11632.707749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11636.777842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11639.079184] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11647.393971] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11662.306389] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11662.308596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11662.320366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11662.320376] Lustre: Skipped 2 previous similar messages [11666.134038] Lustre: server umount lustre-MDT0000 complete [11667.433966] LustreError: 362217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11667.458933] LustreError: 362217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11669.795219] LustreError: 362951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541452 with bad export cookie 9418515946312005762 [11669.798638] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11669.802015] LustreError: 362951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11670.021216] Lustre: server umount lustre-MDT0001 complete [11684.391364] Lustre: server umount lustre-OST0000 complete [11702.296372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11702.967475] LustreError: 366125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11703.063273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11708.050313] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11717.276295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11722.789072] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11731.020610] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11737.327376] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11745.096905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11745.410654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11745.415305] Lustre: Skipped 1 previous similar message [11747.437995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11751.781627] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11761.521458] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11768.810864] Lustre: server umount lustre-OST0000 complete [11778.530268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11778.550454] Lustre: Skipped 3 previous similar messages [11778.552739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11778.552747] Lustre: Skipped 2 previous similar messages [11783.650137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11783.658094] Lustre: Skipped 1 previous similar message [11785.817979] Lustre: server umount lustre-MDT0000 complete [11788.775354] LustreError: 366125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11788.806517] LustreError: 366125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11789.792753] Lustre: 357814:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788541556/real 1788541556] req@0000000059de3f36 x1875421443483904/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788541572 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11789.798146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11789.803334] Lustre: 357814:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11791.533524] LustreError: 367972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541574 with bad export cookie 9418515946312006840 [11791.534051] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11791.542548] LustreError: 367972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11791.895971] Lustre: server umount lustre-MDT0001 complete [11803.673191] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [11806.498378] Key type lgssc unregistered [11806.818203] LNet: 369658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11806.829691] LNetError: 369658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11807.850511] LNet: Removed LNI 192.168.202.126@tcp [11808.836830] Key type .llcrypt unregistered [11808.839351] Key type ._llcrypt unregistered [11834.515594] Key type ._llcrypt registered [11834.518429] Key type .llcrypt registered [11834.643260] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [11836.178828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11836.205454] alg: No test for adler32 (adler32-zlib) [11837.350240] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [11837.664267] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [11839.392361] Key type lgssc registered [11840.748097] Lustre: Echo OBD driver; http://www.lustre.org/ [11849.085595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11849.103464] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11849.115102] LustreError: 371102:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11849.141706] LustreError: 371102:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11849.152946] LustreError: 371102:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11849.159967] LustreError: 371102:0:(tgt_mount.c:2664:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11849.168597] LustreError: 371102:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11857.553807] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11857.560875] LustreError: 371499:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11857.582479] LustreError: 371499:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11857.588447] LustreError: 371499:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11857.595713] LustreError: 371499:0:(tgt_mount.c:2664:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11857.609832] LustreError: 371499:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11880.539325] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [11883.949183] Key type lgssc unregistered [11884.453647] LNet: 372502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11884.459420] LNetError: 372502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11884.493871] LNet: Removed LNI 192.168.202.126@tcp [11885.403536] Key type .llcrypt unregistered [11885.405138] Key type ._llcrypt unregistered [11905.540879] Key type ._llcrypt registered [11905.544250] Key type .llcrypt registered [11905.677810] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [11920.976184] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [11922.005360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11922.019143] alg: No test for adler32 (adler32-zlib) [11923.274883] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [11923.624382] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [11925.256376] Key type lgssc registered [11926.338748] Lustre: Echo OBD driver; http://www.lustre.org/ [11963.537753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11963.573047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11964.914798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11964.950291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11965.050320] Lustre: lustre-MDT0000: new disk, initializing [11965.130487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11965.181631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11971.229026] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11983.421512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11983.532224] Lustre: 375655:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11983.620430] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11983.630910] Lustre: Skipped 1 previous similar message [11983.727466] Lustre: lustre-MDT0001: new disk, initializing [11983.821100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11983.858884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11983.874321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11989.626866] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11999.557209] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12005.419611] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12013.418768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12013.644117] Lustre: lustre-OST0000: new disk, initializing [12013.646927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12013.650920] Lustre: 377621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12013.722636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12021.313582] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12023.994180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12024.016596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12024.211101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12032.280059] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12038.010450] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12038.223598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12043.994498] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12044.231671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12049.894287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12049.895946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12049.898723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12049.921630] Lustre: Skipped 1 previous similar message [12049.938042] LustreError: Skipped 1 previous similar message [12055.028490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12055.034658] Lustre: Skipped 2 previous similar messages [12060.142469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12061.152332] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12061.285206] Lustre: server umount lustre-OST0000 complete [12065.763716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12065.769440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12065.782774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12065.796741] Lustre: Skipped 1 previous similar message [12070.370653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12070.371371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12071.988365] Lustre: server umount lustre-MDT0000 complete [12076.792288] LustreError: 375647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541859 with bad export cookie 10961899291135283564 [12076.793216] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12076.808278] LustreError: 375647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12077.381639] Lustre: server umount lustre-MDT0001 complete [12087.914955] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12090.453870] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:11:11 (1788541871) [12097.570064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12098.079387] LustreError: 380086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12098.095196] LustreError: 380086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12098.156429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12103.326881] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12103.651956] LustreError: 380087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12107.756479] LustreError: 380086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12112.813639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12112.870727] LustreError: 380087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12113.234542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12117.923726] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12127.970716] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12135.277738] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12142.817336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12143.100211] LustreError: 381911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12143.288874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12150.273967] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12158.503470] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12167.930769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12168.136408] Lustre: lustre-OST0001: new disk, initializing [12168.144098] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12168.155309] Lustre: 382928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12168.294775] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12174.710297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12174.724242] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12174.906440] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12177.361746] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12191.249204] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12198.736237] Lustre: lustre-OST0001: Not available for connect from 192.168.202.26@tcp (stopping) [12198.747867] Lustre: Skipped 1 previous similar message [12200.417189] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12200.442367] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12200.488397] Lustre: Skipped 1 previous similar message [12202.115747] Lustre: server umount lustre-OST0001 complete [12203.814540] LustreError: 381913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12220.902878] LustreError: 381913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12220.909281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12220.925832] LustreError: 381913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [12220.938093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12220.950235] Lustre: Skipped 2 previous similar messages [12225.506836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12225.513698] LustreError: Skipped 1 previous similar message [12235.232671] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12235.444911] Lustre: server umount lustre-OST0000 complete [12241.378269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12241.385258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12241.400133] Lustre: Skipped 2 previous similar messages [12246.340441] Lustre: server umount lustre-MDT0000 complete [12250.574423] LustreError: 380822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542033 with bad export cookie 10961899291135284495 [12250.575732] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12250.595238] LustreError: 380822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12257.254943] Lustre: server umount lustre-MDT0001 complete [12269.433685] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [12272.053648] Key type lgssc unregistered [12272.380934] LNet: 384917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12272.392570] LNetError: 384917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12273.458530] LNet: Removed LNI 192.168.202.126@tcp [12274.323042] Key type .llcrypt unregistered [12274.329643] Key type ._llcrypt unregistered [12328.508205] Key type ._llcrypt registered [12328.510950] Key type .llcrypt registered [12328.624963] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [12329.550834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12329.580125] alg: No test for adler32 (adler32-zlib) [12330.601280] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [12330.873764] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [12332.672863] Key type lgssc registered [12334.105141] Lustre: Echo OBD driver; http://www.lustre.org/ [12343.066318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12343.097762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12344.450183] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12344.474504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12344.712486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12350.132874] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12361.432955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12361.510935] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12361.569301] Lustre: 387712:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12361.864435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12367.005802] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12375.654612] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12383.117115] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12387.297415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12387.307170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12387.348836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12390.372947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12390.374376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12392.559069] Lustre: server umount lustre-MDT0000 complete [12396.789674] LustreError: 388456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542179 with bad export cookie 17449167307556555279 [12396.796959] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12396.811073] LustreError: 388456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12397.063902] Lustre: server umount lustre-MDT0001 complete [12408.717758] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:16:29 (1788542189) [12417.090625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12417.685267] LustreError: 390227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12417.701953] LustreError: 390227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12417.764780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12423.122389] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12423.138978] LustreError: 390228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12428.258974] LustreError: 390227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12433.127549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12433.508642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12438.365487] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12447.077312] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12453.877924] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12461.408217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12461.574047] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12461.780741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12468.768941] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12477.097268] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12485.266562] Lustre: Permanently deactivating lustre-OST0000 [12485.292201] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12492.999571] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12495.124518] LustreError: 393220:0:(mgs_llog.c:2049:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12495.138556] LustreError: 393220:0:(mgs_llog.c:2104:mgs_replace_nids()) Only MGS is allowed to be started [12495.142383] LustreError: 393220:0:(mgs_handler.c:1257:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12498.623301] LustreError: 393269:0:(mgs_llog.c:2104:mgs_replace_nids()) Only MGS is allowed to be started [12498.629545] LustreError: 393269:0:(mgs_handler.c:1257:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12515.296655] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12515.492282] Lustre: server umount lustre-OST0000 complete [12519.247980] LustreError: 393520:0:(mgs_llog.c:2104:mgs_replace_nids()) Only MGS is allowed to be started [12519.255173] LustreError: 393520:0:(mgs_handler.c:1257:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12523.501037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12523.539370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12523.546176] Lustre: Skipped 1 previous similar message [12528.610579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12528.620249] Lustre: Skipped 1 previous similar message [12533.730502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12533.739223] Lustre: Skipped 1 previous similar message [12536.288146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12536.664777] Lustre: server umount lustre-MDT0000 complete [12538.852291] LustreError: 390223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12538.881538] LustreError: 390223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12541.071795] LustreError: 390208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542323 with bad export cookie 17449167307556555860 [12541.084176] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12550.855556] Lustre: server umount lustre-MDT0001 complete [12560.639648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12567.117056] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12570.948207] LustreError: 394679:0:(mgs_handler.c:1257:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12579.700873] Lustre: 395018:0:(mgs_llog.c:1814:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.126@tcp [12582.664297] Lustre: server umount MGS complete [12595.749691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12596.593806] LustreError: 395743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12596.619989] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12596.623971] Lustre: Skipped 1 previous similar message [12596.699488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12602.166050] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12606.946049] LustreError: 395744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12606.980582] LustreError: 395744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12611.807543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12612.076246] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12612.156759] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12617.610388] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12625.672204] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12632.004703] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12634.598769] Lustre: Permanently reactivating lustre-OST0000 [12634.610640] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12634.614486] Lustre: Skipped 2 previous similar messages [12664.805592] LustreError: 397572:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12671.077546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12671.335464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12671.341458] Lustre: Skipped 1 previous similar message [12678.050802] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12686.124896] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12694.972230] Lustre: server umount lustre-OST0000 complete [12700.130473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12700.132701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12700.149852] Lustre: Skipped 2 previous similar messages [12700.165433] Lustre: Skipped 1 previous similar message [12709.588708] Lustre: server umount lustre-MDT0000 complete [12710.242068] Lustre: 387034:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788542477/real 1788542477] req@0000000076756270 x1875422334789760/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788542493 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12710.273503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12710.290310] LustreError: 395738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12710.312945] LustreError: 395738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12711.331787] Lustre: 387037:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788542477/real 1788542477] req@00000000e012ce13 x1875422334789504/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788542493 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12713.851459] LustreError: 395723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542496 with bad export cookie 17449167307556556994 [12713.853258] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12713.864215] LustreError: 395723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12714.425854] Lustre: server umount lustre-MDT0001 complete [12729.069853] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [12732.299194] Key type lgssc unregistered [12732.579985] LNet: 400155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12732.587248] LNetError: 400155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12732.613387] LNet: Removed LNI 192.168.202.126@tcp [12733.483977] Key type .llcrypt unregistered [12733.486421] Key type ._llcrypt unregistered [12754.633295] Key type ._llcrypt registered [12754.638783] Key type .llcrypt registered [12754.806637] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [12773.488133] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [12774.626687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12774.775899] alg: No test for adler32 (adler32-zlib) [12775.854362] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [12776.215329] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [12778.000423] Key type lgssc registered [12779.069276] Lustre: Echo OBD driver; http://www.lustre.org/ [12819.028156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12819.049142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12820.197526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12820.214199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12820.264164] Lustre: lustre-MDT0000: new disk, initializing [12820.306902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12820.318431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12824.572736] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12836.578231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12836.695313] Lustre: 403315:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12836.827340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12836.830495] Lustre: Skipped 1 previous similar message [12836.982729] Lustre: lustre-MDT0001: new disk, initializing [12837.187623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12837.231055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12837.243650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12842.422520] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12853.492919] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12861.549482] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12869.595405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12869.912503] Lustre: lustre-OST0000: new disk, initializing [12869.916538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12869.923023] Lustre: 405285:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12870.065930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12876.950200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12876.967596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12877.120643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12877.675538] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12888.900247] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12894.833342] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12895.130951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12901.529928] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12901.930721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12908.006028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12908.013872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12908.019284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12910.605066] Lustre: server umount lustre-OST0000 complete [12918.254275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12918.276205] Lustre: Skipped 1 previous similar message [12918.286169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12918.299793] Lustre: Skipped 2 previous similar messages [12921.757890] Lustre: server umount lustre-MDT0000 complete [12926.393340] LustreError: 405284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542709 with bad export cookie 1341373521882672701 [12926.394980] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12926.403771] LustreError: 405284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12926.903129] Lustre: server umount lustre-MDT0001 complete [12936.667873] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12939.096604] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:25:19 (1788542719) [12946.253256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12946.738136] LustreError: 407745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12946.760232] LustreError: 407745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12946.909191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12952.033481] LustreError: 407746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12952.106532] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12957.168329] LustreError: 407745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12961.607182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12962.136961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12966.915422] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12976.410744] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12984.858177] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12995.314747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12995.837994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13003.037847] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13011.792485] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13013.921576] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13017.835309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13026.814381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13026.826918] LustreError: Skipped 1 previous similar message [13026.838050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13026.862863] Lustre: Skipped 1 previous similar message [13026.866906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13026.879457] Lustre: Skipped 1 previous similar message [13029.701474] Lustre: server umount lustre-OST0000 complete [13033.956897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13033.965328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13033.980274] Lustre: Skipped 1 previous similar message [13033.986405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13040.172406] Lustre: server umount lustre-MDT0000 complete [13042.150587] LustreError: 407746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13042.177648] LustreError: 407746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13044.570379] LustreError: 409591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542827 with bad export cookie 1341373521882673625 [13044.570379] LustreError: 407726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542827 with bad export cookie 1341373521882673625 [13044.570646] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13045.112339] Lustre: server umount lustre-MDT0001 complete [13057.641873] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [13060.327397] Key type lgssc unregistered [13060.614736] LNet: 411475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13060.622510] LNetError: 411475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13060.635928] LNet: Removed LNI 192.168.202.126@tcp [13061.392459] Key type .llcrypt unregistered [13061.395143] Key type ._llcrypt unregistered [13088.132775] Key type ._llcrypt registered [13088.135413] Key type .llcrypt registered [13088.279090] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [13088.606289] Key type .llcrypt unregistered [13088.608892] Key type ._llcrypt unregistered [13105.896441] Key type ._llcrypt registered [13105.899947] Key type .llcrypt registered [13106.002323] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [13107.197030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13107.379287] alg: No test for adler32 (adler32-zlib) [13108.556326] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [13108.783046] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [13110.424230] Key type lgssc registered [13111.598506] Lustre: Echo OBD driver; http://www.lustre.org/ [13119.859373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13119.883232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13121.412906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13125.160270] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13131.753549] LustreError: 413684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13131.779522] LustreError: 413684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13133.777362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13134.221621] LustreError: 413685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13134.246245] LustreError: 413685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13134.256743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13139.429194] LustreError: 413681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13139.452948] LustreError: 413681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13139.908705] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13144.548334] LustreError: 413684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13144.570853] LustreError: 413684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13146.615900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13146.649253] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13151.714928] LustreError: 413685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13151.730443] LustreError: 413685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13153.318804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13153.676494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13158.359693] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13167.012089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13167.025644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13168.837941] Lustre: server umount lustre-OST0000 complete [13173.272201] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13173.278283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13173.286780] Lustre: Skipped 1 previous similar message [13173.291161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13173.298221] Lustre: Skipped 1 previous similar message [13173.587627] Lustre: server umount lustre-MDT0000 complete [13177.315679] LustreError: 415421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13177.326118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13178.644074] LustreError: 413665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542961 with bad export cookie 9558077818059646140 [13178.650741] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13178.659360] LustreError: 413665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13179.037182] Lustre: server umount lustre-MDT0001 complete [13193.242887] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [13196.904543] Key type lgssc unregistered [13197.222083] LNet: 416831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13197.231766] LNetError: 416831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13197.248960] LNet: Removed LNI 192.168.202.126@tcp [13198.214940] Key type .llcrypt unregistered [13198.217125] Key type ._llcrypt unregistered [13229.941940] Key type ._llcrypt registered [13229.947111] Key type .llcrypt registered [13230.122539] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [13232.400229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13232.490521] alg: No test for adler32 (adler32-zlib) [13233.499594] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [13233.802560] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [13235.472195] Key type lgssc registered [13236.627433] Lustre: Echo OBD driver; http://www.lustre.org/ [13246.164858] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13246.194733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13247.414036] Lustre: 418230:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [13262.944883] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13268.064345] LustreError: 418218:0:(mgc_request.c:317:do_config_log_add()) MGC192.168.202.126@tcp: failed processing log, type 4: rc = -110 [13272.548766] Lustre: 418232:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [13278.368527] Lustre: 418218:0:(mgc_request.c:2298:mgc_process_log()) MGC192.168.202.126@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [13288.624940] Lustre: 418230:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [13289.504309] Lustre: 418234:0:(mgc_request.c:2298:mgc_process_log()) MGC192.168.202.126@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [13294.050140] Lustre: 418232:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [13298.848221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13299.170907] Lustre: 418230:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [13299.747410] LustreError: 418234:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [13306.462880] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13309.415282] Lustre: 418230:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [13309.441240] Lustre: 418230:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [13316.186226] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13320.225712] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13323.461308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13324.357919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13324.424692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13329.380942] LustreError: 418248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13329.408245] LustreError: 418248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13330.320279] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13333.490419] LustreError: 419263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13333.508563] LustreError: 419263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13338.594716] LustreError: 419264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13338.629021] LustreError: 419264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13343.720800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13343.725807] LustreError: 419263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13343.759100] LustreError: 419263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13344.624212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13351.896354] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13362.890994] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13370.809699] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13380.582152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13380.586159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13380.607061] Lustre: Skipped 1 previous similar message [13380.618685] Lustre: Skipped 1 previous similar message [13385.712801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13385.722922] Lustre: Skipped 1 previous similar message [13386.249188] Lustre: server umount lustre-OST0000 complete [13390.817469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13390.825153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13390.839848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13390.849405] Lustre: Skipped 1 previous similar message [13390.866546] Lustre: Skipped 1 previous similar message [13395.938329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13395.946511] Lustre: Skipped 1 previous similar message [13396.881357] Lustre: server umount lustre-MDT0000 complete [13401.062216] LustreError: 419268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13401.091968] LustreError: 419268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13401.309403] LustreError: 418224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788543184 with bad export cookie 681947954093762242 [13401.313284] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13401.320525] LustreError: 418224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13401.672031] Lustre: server umount lustre-MDT0001 complete [13414.168646] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [13417.508948] Key type lgssc unregistered [13417.913445] LNet: 421797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13417.922578] LNetError: 421797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13418.984062] LNet: Removed LNI 192.168.202.126@tcp [13420.052144] Key type .llcrypt unregistered [13420.056053] Key type ._llcrypt unregistered [13447.297963] Key type ._llcrypt registered [13447.301822] Key type .llcrypt registered [13447.418213] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [13448.775355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13448.907474] alg: No test for adler32 (adler32-zlib) [13450.253034] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [13450.636617] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [13452.312211] Key type lgssc registered [13453.713028] Lustre: Echo OBD driver; http://www.lustre.org/ [13462.342968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13462.367094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13463.807054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13468.981941] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13473.259802] LustreError: 423218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13473.275271] LustreError: 423218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13478.086421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13478.437606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13484.822481] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13495.219897] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13502.449184] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13511.701974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13512.099893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13520.568846] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13530.330692] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13533.692453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13542.709816] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.26@tcp (stopping) [13543.913941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13543.929609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13543.946150] Lustre: Skipped 2 previous similar messages [13543.966684] Lustre: Skipped 2 previous similar messages [13546.383512] Lustre: server umount lustre-MDT0000 complete [13547.812803] LustreError: 425264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13549.027747] LustreError: 423215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13549.053695] LustreError: 423215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13549.401694] LustreError: 423198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.26@tcp arrived at 1788543332 with bad export cookie 17836909007617903222 [13549.426287] LustreError: 423198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13552.257970] LustreError: 423200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788543335 with bad export cookie 17836909007617902501 [13552.283859] LustreError: 423200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13552.289264] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13554.146066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13554.150693] LustreError: 423218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13554.153879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13554.159896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13554.223698] LustreError: 423218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13557.153859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13558.609640] Lustre: server umount lustre-OST0000 complete [13562.280482] LustreError: 423218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13562.316670] LustreError: 423218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13567.568537] Lustre: server umount lustre-MDT0001 complete [13583.049519] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [13586.552963] Key type lgssc unregistered [13586.883370] LNet: 426904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13586.889401] LNetError: 426904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13586.914921] LNet: Removed LNI 192.168.202.126@tcp [13587.856191] Key type .llcrypt unregistered [13587.858823] Key type ._llcrypt unregistered [13617.759389] Key type ._llcrypt registered [13617.762420] Key type .llcrypt registered [13617.864890] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [13619.028401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13619.115423] alg: No test for adler32 (adler32-zlib) [13620.335117] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [13620.650759] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [13622.392207] Key type lgssc registered [13623.920582] Lustre: Echo OBD driver; http://www.lustre.org/ [13632.692599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13632.706857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13634.294512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13639.445736] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13639.664817] LustreError: 428317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13639.691807] LustreError: 428317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13644.784023] LustreError: 428318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13645.814312] LustreError: 428317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13650.258241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13650.669274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13656.328651] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13665.848873] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13673.167665] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13681.151615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13681.670329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13689.280506] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13698.137181] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13706.232177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13715.279196] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.26@tcp (stopping) [13716.459733] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13716.465253] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13716.489971] Lustre: Skipped 1 previous similar message [13716.532981] Lustre: Skipped 1 previous similar message [13719.766390] Lustre: server umount lustre-MDT0001 complete [13720.351807] LustreError: 430159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13726.689645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13726.692762] LustreError: 428317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13726.712865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13726.732279] LustreError: 428317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13731.809961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13732.905962] Lustre: server umount lustre-OST0000 complete [13736.937446] LustreError: 428317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13736.981811] LustreError: 428317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13737.602185] Lustre: server umount lustre-MDT0000 complete [13756.420559] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [13760.196946] Key type lgssc unregistered [13760.676411] LNet: 432027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13760.690964] LNetError: 432027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13760.706532] LNet: Removed LNI 192.168.202.126@tcp [13761.700739] Key type .llcrypt unregistered [13761.703301] Key type ._llcrypt unregistered [13796.383691] Key type ._llcrypt registered [13796.386489] Key type .llcrypt registered [13796.529365] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [13815.645783] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [13817.182381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13817.275825] alg: No test for adler32 (adler32-zlib) [13818.443093] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [13818.634699] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [13820.336856] Key type lgssc registered [13821.377232] Lustre: Echo OBD driver; http://www.lustre.org/ [13871.356186] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [13893.354043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13893.404031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13894.816579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13894.838208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13894.938059] Lustre: lustre-MDT0000: new disk, initializing [13895.054533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13895.073533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13900.957084] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13921.027665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13921.412160] Lustre: lustre-OST0000: new disk, initializing [13921.416228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13921.423616] Lustre: Skipped 1 previous similar message [13921.428490] Lustre: 437062:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13921.602192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13927.142674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13927.160880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13927.206674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13930.270365] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13941.561206] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13958.918690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13959.060659] Lustre: 436046:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13959.156172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13959.219409] Lustre: lustre-MDT0001: new disk, initializing [13959.397256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13959.452191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13959.465469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13967.050958] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13979.550570] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13979.815368] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13985.877735] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13986.057576] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13991.912457] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13991.919564] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13991.924055] Lustre: Skipped 1 previous similar message [13991.938390] Lustre: Skipped 1 previous similar message [13991.944423] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13994.325596] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.26@tcp (stopping) [13997.029436] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13997.042085] Lustre: Skipped 1 previous similar message [13999.409627] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.26@tcp (stopping) [14004.510274] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.26@tcp (stopping) [14004.519194] Lustre: Skipped 2 previous similar messages [14006.240120] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14006.467877] Lustre: server umount lustre-MDT0001 complete [14012.388614] LustreError: 436056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14012.423233] LustreError: 436056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [14017.521610] LustreError: 437069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14022.631754] LustreError: 436052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14022.662384] LustreError: 436052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14027.752795] LustreError: 436055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14027.753249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14027.806609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14032.881216] LustreError: 436056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14033.286263] Lustre: server umount lustre-OST0000 complete [14044.067437] Lustre: server umount lustre-MDT0000 complete [14063.430727] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [14066.384801] Key type lgssc unregistered [14066.729926] LNet: 440812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14066.746024] LNetError: 440812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14067.819878] LNet: Removed LNI 192.168.202.126@tcp [14069.083716] Key type .llcrypt unregistered [14069.088065] Key type ._llcrypt unregistered [14120.041903] Key type ._llcrypt registered [14120.044930] Key type .llcrypt registered [14120.174965] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [14120.665757] Key type .llcrypt unregistered [14120.667822] Key type ._llcrypt unregistered [14142.102739] Key type ._llcrypt registered [14142.110764] Key type .llcrypt registered [14142.303519] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [14143.791062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14144.056764] alg: No test for adler32 (adler32-zlib) [14145.511564] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [14146.036064] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [14147.752232] Key type lgssc registered [14149.337116] Lustre: Echo OBD driver; http://www.lustre.org/ [14186.879359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14186.922733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14188.350928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14188.405815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14188.514516] Lustre: lustre-MDT0000: new disk, initializing [14188.816674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14188.862539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14195.210304] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [14208.826506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14208.969301] Lustre: 444187:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14209.047559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14209.052419] Lustre: Skipped 1 previous similar message [14209.127502] Lustre: lustre-MDT0001: new disk, initializing [14209.253866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14209.308401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14209.322420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14216.243559] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [14228.394350] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14237.619699] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14247.786916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14248.285722] Lustre: lustre-OST0000: new disk, initializing [14248.298672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14248.310037] Lustre: 446162:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14248.466726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14256.921703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14256.948062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14257.150050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14257.460140] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [14270.099199] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14281.200072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14281.219944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14281.238273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14282.738620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14282.747829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14282.780783] Lustre: Skipped 1 previous similar message [14285.463727] Lustre: server umount lustre-MDT0000 complete [14290.126118] LustreError: 444182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788544072 with bad export cookie 9950087751008944089 [14290.128022] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14290.144277] LustreError: 444182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14290.825786] Lustre: server umount lustre-MDT0001 complete [14306.438349] Lustre: server umount lustre-OST0000 complete [14319.460611] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:48:20 (1788544100) [14330.031288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14330.897767] LustreError: 448192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14330.925735] LustreError: 448192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14331.012532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14336.491244] LustreError: 448193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14337.064135] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [14340.586351] LustreError: 448192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14345.729171] LustreError: 448193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14348.408334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14348.881412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14354.729134] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [14364.147222] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14371.494966] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14393.283786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14393.553349] Lustre: Found index 0 for lustre-OST0000, updating log [14393.923348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14401.015165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14401.313103] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [14410.760904] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14419.428315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14419.435670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14419.464633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14419.474259] Lustre: Skipped 2 previous similar messages [14423.772669] Lustre: server umount lustre-OST0000 complete [14440.938792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14440.963334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14440.998111] Lustre: Skipped 1 previous similar message [14441.017552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14441.020222] Lustre: Skipped 3 previous similar messages [14447.082291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14447.089538] Lustre: Skipped 3 previous similar messages [14455.264229] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14455.666183] Lustre: server umount lustre-MDT0000 complete [14458.338304] LustreError: 448193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14458.370236] LustreError: 448193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14460.524241] LNet: 1 local NIs in recovery (showing 1): 192.168.202.126@tcp [14460.646811] LustreError: 448174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788544243 with bad export cookie 9950087751008945524 [14460.657747] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14460.669301] LustreError: 448174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14461.226454] Lustre: server umount lustre-MDT0001 complete [14473.722940] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:50:54 (1788544254) [15100.472534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15101.260430] LustreError: 454620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15101.408429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15106.253986] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15106.530404] LustreError: 454621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15111.669273] LustreError: 454620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15115.177557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15115.428825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15120.297139] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15128.031980] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15133.870159] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15151.776162] LNet: 1 local NIs in recovery (showing 1): 192.168.202.126@tcp [15154.865268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15155.287849] Lustre: Found index 0 for lustre-OST0000, updating log [15155.743954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15162.131721] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15171.010159] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15173.350515] Lustre: server umount lustre-OST0000 complete [15178.351921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15178.363369] Lustre: Skipped 2 previous similar messages [15178.368453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15178.385061] Lustre: Skipped 5 previous similar messages [15179.426444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15179.438169] Lustre: Skipped 1 previous similar message [15181.474993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15181.484139] Lustre: Skipped 3 previous similar messages [15183.524368] LustreError: 454620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15183.551900] LustreError: 454620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15183.850361] Lustre: server umount lustre-MDT0000 complete [15187.699815] LustreError: 454600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788544970 with bad export cookie 9950087751008946665 [15187.702259] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15187.707482] LustreError: 454600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15188.074735] Lustre: server umount lustre-MDT0001 complete [15212.192242] LNet: 1 local NIs in recovery (showing 1): 192.168.202.126@tcp [15220.583942] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [15230.763609] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [15275.906455] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:04:16 (1788545056) [15296.903474] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [15300.439751] Key type lgssc unregistered [15300.908593] LNet: 461238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15300.915665] LNetError: 461238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15418.741227] LNet: Removed LNI 192.168.202.126@tcp [15419.769357] Key type .llcrypt unregistered [15419.775912] Key type ._llcrypt unregistered [15439.637218] Key type ._llcrypt registered [15439.642841] Key type .llcrypt registered [15439.796797] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [15441.305817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15441.346228] alg: No test for adler32 (adler32-zlib) [15442.589048] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [15442.948808] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [15444.656187] Key type lgssc registered [15446.068905] Lustre: Echo OBD driver; http://www.lustre.org/ [15450.674977] LNet: Added LNI 192.168.202.126@tcp100 [8/256/0/180] [15451.659299] LNet: Added LNI 192.168.202.126@tcp101 [8/256/0/180] [15452.717803] LNet: Added LNI 192.168.202.126@tcp102 [8/256/0/180] [15453.841709] LNet: Added LNI 192.168.202.126@tcp103 [8/256/0/180] [15455.011043] LNet: Added LNI 192.168.202.126@tcp104 [8/256/0/180] [15456.101597] LNet: Added LNI 192.168.202.126@tcp105 [8/256/0/180] [15457.109088] LNet: Added LNI 192.168.202.126@tcp106 [8/256/0/180] [15458.332972] LNet: Added LNI 192.168.202.126@tcp107 [8/256/0/180] [15459.543651] LNet: Added LNI 192.168.202.126@tcp108 [8/256/0/180] [15460.677449] LNet: Added LNI 192.168.202.126@tcp109 [8/256/0/180] [15461.885933] LNet: Added LNI 192.168.202.126@tcp110 [8/256/0/180] [15462.970853] LNet: Added LNI 192.168.202.126@tcp111 [8/256/0/180] [15464.111920] LNet: Added LNI 192.168.202.126@tcp112 [8/256/0/180] [15465.097761] LNet: Added LNI 192.168.202.126@tcp113 [8/256/0/180] [15466.101877] LNet: Added LNI 192.168.202.126@tcp114 [8/256/0/180] [15467.159727] LNet: Added LNI 192.168.202.126@tcp115 [8/256/0/180] [15468.096497] LNet: Added LNI 192.168.202.126@tcp116 [8/256/0/180] [15469.203423] LNet: Added LNI 192.168.202.126@tcp117 [8/256/0/180] [15470.361460] LNet: Added LNI 192.168.202.126@tcp118 [8/256/0/180] [15471.554959] LNet: Added LNI 192.168.202.126@tcp119 [8/256/0/180] [15472.729230] LNet: Added LNI 192.168.202.126@tcp120 [8/256/0/180] [15474.098365] LNet: Added LNI 192.168.202.126@tcp121 [8/256/0/180] [15475.422981] LNet: Added LNI 192.168.202.126@tcp122 [8/256/0/180] [15476.318825] LNet: Added LNI 192.168.202.126@tcp123 [8/256/0/180] [15477.196795] LNet: Added LNI 192.168.202.126@tcp124 [8/256/0/180] [15478.425125] LNet: Added LNI 192.168.202.126@tcp125 [8/256/0/180] [15479.249697] LNet: Added LNI 192.168.202.126@tcp126 [8/256/0/180] [15480.328754] LNet: Added LNI 192.168.202.126@tcp127 [8/256/0/180] [15481.386987] LNet: Added LNI 192.168.202.126@tcp128 [8/256/0/180] [15482.477572] LNet: Added LNI 192.168.202.126@tcp129 [8/256/0/180] [15483.431163] LNet: Added LNI 192.168.202.126@tcp130 [8/256/0/180] [15484.338048] LNet: Added LNI 192.168.202.126@tcp131 [8/256/0/180] [15485.461389] LNet: Added LNI 192.168.202.126@tcp132 [8/256/0/180] [15486.416350] LNet: Added LNI 192.168.202.126@tcp133 [8/256/0/180] [15487.421678] LNet: Added LNI 192.168.202.126@tcp134 [8/256/0/180] [15488.425067] LNet: Added LNI 192.168.202.126@tcp135 [8/256/0/180] [15496.184871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15496.233045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15497.448493] Lustre: 464267:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.126@tcp: can't add NID 192.168.202.126@tcp130: rc = -75 [15497.534107] LNetError: 464292:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [15497.552074] LNetError: Protocol error connecting to 192.168.202.126@tcp116 at host 192.168.202.126:988: is it running a compatible version of Lustre? [15497.578110] LNetError: Skipped 1 previous similar message [15497.649882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15497.682642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15497.779156] Lustre: lustre-MDT0000: new disk, initializing [15497.865828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15497.884939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15498.594815] LNetError: 464292:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [15498.620331] LNetError: 464292:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15498.637408] LNetError: Protocol error connecting to 192.168.202.126@tcp104 at host 192.168.202.126:988: is it running a compatible version of Lustre? [15503.596658] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15515.146023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15515.238764] Lustre: 464285:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15515.319634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15515.323429] Lustre: Skipped 1 previous similar message [15515.394821] Lustre: lustre-MDT0001: new disk, initializing [15515.422534] Lustre: 465091:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.126@tcp: can't add NID 192.168.202.126@tcp130: rc = -75 [15515.428393] Lustre: 465091:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15515.526646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15515.562431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15515.566278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15521.513483] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15523.379893] Lustre: 465424:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.126@tcp: can't add NID 192.168.202.126@tcp130: rc = -75 [15523.395487] Lustre: 465424:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15530.023520] LNetError: 464289:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [15530.050845] LNetError: Protocol error connecting to 192.168.202.126@tcp116 at host 192.168.202.126:988: is it running a compatible version of Lustre? [15533.252118] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15534.120768] LNetError: 461724:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [15534.145576] LNetError: Protocol error connecting to 192.168.202.126@tcp122 at host 192.168.202.126:988: is it running a compatible version of Lustre? [15540.060765] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15543.330188] LNetError: 461723:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [15543.350333] LNetError: 461723:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15543.357913] LNetError: Protocol error connecting to 192.168.202.126@tcp116 at host 192.168.202.126:988: is it running a compatible version of Lustre? [15543.372501] LNetError: Skipped 1 previous similar message [15547.315794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15547.636449] Lustre: lustre-OST0000: new disk, initializing [15547.646556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15547.656424] Lustre: 466263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15547.812292] Lustre: 466268:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.126@tcp: can't add NID 192.168.202.126@tcp130: rc = -75 [15547.825668] Lustre: 466268:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15547.896219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15553.235210] Lustre: 466536:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.126@tcp: can't add NID 192.168.202.126@tcp130: rc = -75 [15553.247843] Lustre: 466536:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15553.351024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15553.367156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15553.548283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15554.949125] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15556.645708] LNetError: 464288:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [15556.658479] LNetError: 464288:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15556.673803] LNetError: Protocol error connecting to 192.168.202.126@tcp116 at host 192.168.202.126:988: is it running a compatible version of Lustre? [15556.703175] LNetError: Skipped 2 previous similar messages [15565.969828] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15573.092364] LNetError: 464292:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [15573.101164] LNetError: 464292:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [15573.107056] LNetError: Protocol error connecting to 192.168.202.126@tcp122 at host 192.168.202.126:988: is it running a compatible version of Lustre? [15573.115207] LNetError: Skipped 3 previous similar messages [15573.987809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15573.992121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15574.008508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15574.019584] Lustre: Skipped 1 previous similar message [15579.111656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15579.114844] Lustre: Skipped 1 previous similar message [15579.863323] Lustre: server umount lustre-OST0000 complete [15584.226024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15584.251502] Lustre: Skipped 1 previous similar message [15584.263284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15584.270648] Lustre: Skipped 1 previous similar message [15589.347852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15589.359167] Lustre: Skipped 1 previous similar message [15590.651633] Lustre: server umount lustre-MDT0000 complete [15594.743474] LustreError: 464275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788545377 with bad export cookie 11259120155023080777 [15594.754139] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15594.759014] LustreError: 464275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [15595.509114] Lustre: server umount lustre-MDT0001 complete [15607.089405] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [15609.783972] Key type lgssc unregistered [15610.159260] LNet: 468176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15610.170857] LNetError: 468176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15610.177941] LNet: Removed LNI 192.168.202.126@tcp [15611.236374] LNet: Removed LNI 192.168.202.126@tcp100 [15611.243741] LNetError: 468176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15611.257767] LNetError: 468176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15611.275783] LNet: Removed LNI 192.168.202.126@tcp101 [15612.320190] LNet: Removed LNI 192.168.202.126@tcp102 [15612.322391] LNetError: 468176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15612.333085] LNetError: 468176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15612.342171] LNet: Removed LNI 192.168.202.126@tcp103 [15613.408222] LNet: Removed LNI 192.168.202.126@tcp104 [15614.432202] LNet: Removed LNI 192.168.202.126@tcp105 [15614.434409] LNetError: 468176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15614.440132] LNetError: 468176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15614.448275] LNet: Removed LNI 192.168.202.126@tcp106 [15615.458641] LNet: Removed LNI 192.168.202.126@tcp107 [15616.480168] LNet: Removed LNI 192.168.202.126@tcp108 [15617.504555] LNet: Removed LNI 192.168.202.126@tcp109 [15618.528311] LNet: Removed LNI 192.168.202.126@tcp110 [15618.530430] LNetError: 468176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15618.544335] LNetError: 468176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15618.560846] LNet: Removed LNI 192.168.202.126@tcp111 [15619.616559] LNet: Removed LNI 192.168.202.126@tcp112 [15620.640304] LNet: Removed LNI 192.168.202.126@tcp113 [15621.664164] LNet: Removed LNI 192.168.202.126@tcp114 [15622.689257] LNet: Removed LNI 192.168.202.126@tcp115 [15623.713712] LNet: Removed LNI 192.168.202.126@tcp116 [15624.736724] LNet: Removed LNI 192.168.202.126@tcp117 [15625.760132] LNet: Removed LNI 192.168.202.126@tcp118 [15626.784288] LNet: Removed LNI 192.168.202.126@tcp119 [15626.791671] LNetError: 468176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15626.812893] LNetError: 468176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15626.822966] LNet: Removed LNI 192.168.202.126@tcp120 [15627.873566] LNet: Removed LNI 192.168.202.126@tcp121 [15628.896143] LNet: Removed LNI 192.168.202.126@tcp122 [15628.909374] LNet: Removed LNI 192.168.202.126@tcp123 [15628.928167] LNet: Removed LNI 192.168.202.126@tcp124 [15628.939305] LNet: Removed LNI 192.168.202.126@tcp125 [15628.944347] LNet: Removed LNI 192.168.202.126@tcp126 [15628.949429] LNet: Removed LNI 192.168.202.126@tcp127 [15628.953272] LNet: Removed LNI 192.168.202.126@tcp128 [15628.958451] LNet: Removed LNI 192.168.202.126@tcp129 [15628.962442] LNet: Removed LNI 192.168.202.126@tcp130 [15628.971248] LNet: Removed LNI 192.168.202.126@tcp131 [15628.973332] LNet: Removed LNI 192.168.202.126@tcp132 [15628.983589] LNet: Removed LNI 192.168.202.126@tcp133 [15628.989302] LNet: Removed LNI 192.168.202.126@tcp134 [15629.005437] LNet: Removed LNI 192.168.202.126@tcp135 [15630.094535] Key type .llcrypt unregistered [15630.096197] Key type ._llcrypt unregistered [15655.384555] Key type ._llcrypt registered [15655.385689] Key type .llcrypt registered [15655.482279] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [15673.058309] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [15674.945603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15675.155027] alg: No test for adler32 (adler32-zlib) [15676.308425] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [15676.562904] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [15678.296175] Key type lgssc registered [15679.602597] Lustre: Echo OBD driver; http://www.lustre.org/ [15724.947281] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:11:45 (1788545505) [15746.881977] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [15750.756346] Key type lgssc unregistered [15751.175576] LNet: 472268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15751.188540] LNetError: 472268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15751.200676] LNet: Removed LNI 192.168.202.126@tcp [15752.346332] Key type .llcrypt unregistered [15752.353071] Key type ._llcrypt unregistered [15773.690741] Key type ._llcrypt registered [15773.692561] Key type .llcrypt registered [15773.845287] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [15775.119760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15775.303701] alg: No test for adler32 (adler32-zlib) [15776.408151] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [15776.743517] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [15778.409901] Key type lgssc registered [15780.173846] Lustre: Echo OBD driver; http://www.lustre.org/ [15791.315186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15791.360790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15792.503902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15792.524469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15792.583165] Lustre: lustre-MDT0000: new disk, initializing [15792.638935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15792.647747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15797.715909] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15808.014996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15808.106838] Lustre: 473480:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15808.170652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15808.174348] Lustre: Skipped 1 previous similar message [15808.252444] Lustre: lustre-MDT0001: new disk, initializing [15808.367557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15808.412386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15808.432485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15813.666993] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15824.514580] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15831.750486] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15840.714551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15841.011862] Lustre: lustre-OST0000: new disk, initializing [15841.015810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15841.022408] Lustre: 475451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15841.184806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15847.087900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15847.105645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15847.286800] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15848.980495] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15860.307889] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15865.727763] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing lnet_if_list [15871.241421] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15871.263994] LNet: Added LNI 192.168.202.126@tcp10 [8/256/0/180] [15878.240102] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15878.268556] LNet: Added LNI 192.168.202.126@tcp11 [8/256/0/180] [15882.383844] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15882.440194] LNet: Added LNI 192.168.202.126@tcp12 [8/256/0/180] [15889.709206] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15889.760682] LNet: Added LNI 192.168.202.126@tcp13 [8/256/0/180] [15895.391626] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15895.473622] LNet: Added LNI 192.168.202.126@tcp14 [8/256/0/180] [15900.806432] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15900.851945] LNet: Added LNI 192.168.202.126@tcp15 [8/256/0/180] [15906.718557] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15906.744778] LNet: Added LNI 192.168.202.126@tcp16 [8/256/0/180] [15912.121734] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15912.143280] LNet: Added LNI 192.168.202.126@tcp17 [8/256/0/180] [15917.423867] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15917.456979] LNet: Added LNI 192.168.202.126@tcp18 [8/256/0/180] [15923.945542] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15923.978352] LNet: Added LNI 192.168.202.126@tcp19 [8/256/0/180] [15930.770654] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15930.803348] LNet: Added LNI 192.168.202.126@tcp20 [8/256/0/180] [15938.016427] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15938.073652] LNet: Added LNI 192.168.202.126@tcp21 [8/256/0/180] [15945.923423] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15945.962879] LNet: Added LNI 192.168.202.126@tcp22 [8/256/0/180] [15953.981465] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15954.010496] LNet: Added LNI 192.168.202.126@tcp23 [8/256/0/180] [15961.509167] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15961.535166] LNet: Added LNI 192.168.202.126@tcp24 [8/256/0/180] [15968.319373] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15968.356691] LNet: Added LNI 192.168.202.126@tcp25 [8/256/0/180] [15974.718828] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15974.791126] LNet: Added LNI 192.168.202.126@tcp26 [8/256/0/180] [15982.578371] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15982.610577] LNet: Added LNI 192.168.202.126@tcp27 [8/256/0/180] [15991.375419] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15991.418976] LNet: Added LNI 192.168.202.126@tcp28 [8/256/0/180] [15998.939672] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15998.987871] LNet: Added LNI 192.168.202.126@tcp29 [8/256/0/180] [16007.291450] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16007.317615] LNet: Added LNI 192.168.202.126@tcp30 [8/256/0/180] [16014.625311] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16014.644422] LNet: Added LNI 192.168.202.126@tcp31 [8/256/0/180] [16022.494397] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16022.531133] LNet: Added LNI 192.168.202.126@tcp32 [8/256/0/180] [16029.350889] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16029.392029] LNet: Added LNI 192.168.202.126@tcp33 [8/256/0/180] [16037.275906] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16037.337341] LNet: Added LNI 192.168.202.126@tcp34 [8/256/0/180] [16045.621711] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16045.654571] LNet: Added LNI 192.168.202.126@tcp35 [8/256/0/180] [16053.396589] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16053.439729] LNet: Added LNI 192.168.202.126@tcp36 [8/256/0/180] [16061.414122] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16061.456442] LNet: Added LNI 192.168.202.126@tcp37 [8/256/0/180] [16069.012650] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16069.078065] LNet: Added LNI 192.168.202.126@tcp38 [8/256/0/180] [16077.701479] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16077.722702] LNet: Added LNI 192.168.202.126@tcp39 [8/256/0/180] [16086.367469] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16086.417563] LNet: Added LNI 192.168.202.126@tcp40 [8/256/0/180] [16096.573249] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16096.610351] LNet: Added LNI 192.168.202.126@tcp41 [8/256/0/180] [16105.738994] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16105.786107] LNet: Added LNI 192.168.202.126@tcp42 [8/256/0/180] [16114.646392] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16114.681582] LNet: Added LNI 192.168.202.126@tcp43 [8/256/0/180] [16123.955163] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16123.994033] LNet: Added LNI 192.168.202.126@tcp44 [8/256/0/180] [16134.084498] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16134.135912] LNet: Added LNI 192.168.202.126@tcp45 [8/256/0/180] [16143.222083] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16143.247031] LNet: Added LNI 192.168.202.126@tcp46 [8/256/0/180] [16151.425200] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16151.456355] LNet: Added LNI 192.168.202.126@tcp47 [8/256/0/180] [16160.877715] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16160.910598] LNet: Added LNI 192.168.202.126@tcp48 [8/256/0/180] [16170.394885] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16170.424196] LNet: Added LNI 192.168.202.126@tcp49 [8/256/0/180] [16181.022689] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16181.042716] LNet: Added LNI 192.168.202.126@tcp50 [8/256/0/180] [16202.722201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16202.744299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16202.758079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16205.795986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16205.798787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16205.827737] Lustre: Skipped 1 previous similar message [16208.214617] Lustre: server umount lustre-MDT0000 complete [16212.444834] LustreError: 473472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788545995 with bad export cookie 3930512306279167424 [16212.445858] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16212.461656] LustreError: 473472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16212.926426] Lustre: server umount lustre-MDT0001 complete [16231.392619] Lustre: 472750:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788545998/real 1788545998] req@000000000c2f5885 x1875425948287232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788546014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16231.423671] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16231.751243] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [16232.517976] Lustre: 472750:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788545999/real 1788545999] req@000000006ce134d0 x1875425948287360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788546015 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16236.513734] Lustre: 472753:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788546003/real 1788546003] req@00000000d149ac52 x1875425948287488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788546019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16238.561624] Lustre: 472752:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788546005/real 1788546005] req@00000000439c1cc8 x1875425948287744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788546021 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16241.944899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16245.216128] Lustre: 472750:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788546011/real 1788546011] req@00000000d92c6d63 x1875425948287872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788546027 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16258.081815] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x368bfc1e2912424d [16258.095399] Lustre: MGC192.168.202.126@tcp: Connection restored to 0@lo (at 0@lo) [16258.472075] LustreError: 475444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16258.498494] LustreError: 475444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [16258.585737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16260.644285] LustreError: 475445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16260.647834] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16262.694337] LustreError: 475444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16264.849147] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [16267.760560] LustreError: 475445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16272.867496] LustreError: 475444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16276.872681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16277.476081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16282.608774] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16282.942636] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [16292.169790] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16298.287682] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16323.564716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16323.569603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16323.592436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16323.605964] Lustre: Skipped 2 previous similar messages [16327.709507] Lustre: server umount lustre-OST0000 complete [16333.794613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16333.794896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16333.800647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16333.814897] Lustre: Skipped 2 previous similar messages [16338.531247] Lustre: server umount lustre-MDT0000 complete [16338.913193] LustreError: 499712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16338.947683] LustreError: 499712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [16343.207143] LustreError: 473473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788546125 with bad export cookie 3930512306279170637 [16343.212250] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16343.218251] LustreError: 473473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16343.651862] Lustre: server umount lustre-MDT0001 complete [16358.743872] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [16362.673868] Key type lgssc unregistered [16363.200918] LNet: 502513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16363.219273] LNetError: 502513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16363.228269] LNet: Removed LNI 192.168.202.126@tcp [16363.230288] LNet: Removed LNI 192.168.202.126@tcp10 [16363.233967] LNet: Removed LNI 192.168.202.126@tcp11 [16363.237511] LNet: Removed LNI 192.168.202.126@tcp12 [16363.240542] LNet: Removed LNI 192.168.202.126@tcp13 [16363.244769] LNet: Removed LNI 192.168.202.126@tcp14 [16363.263876] LNet: Removed LNI 192.168.202.126@tcp15 [16363.265846] LNet: Removed LNI 192.168.202.126@tcp16 [16363.271850] LNet: Removed LNI 192.168.202.126@tcp17 [16363.274957] LNet: Removed LNI 192.168.202.126@tcp18 [16363.289539] LNet: Removed LNI 192.168.202.126@tcp19 [16363.296967] LNet: Removed LNI 192.168.202.126@tcp20 [16363.300969] LNet: Removed LNI 192.168.202.126@tcp21 [16363.311868] LNet: Removed LNI 192.168.202.126@tcp22 [16363.318983] LNet: Removed LNI 192.168.202.126@tcp23 [16363.325148] LNet: Removed LNI 192.168.202.126@tcp24 [16363.332843] LNet: Removed LNI 192.168.202.126@tcp25 [16363.343149] LNet: Removed LNI 192.168.202.126@tcp26 [16363.348472] LNet: Removed LNI 192.168.202.126@tcp27 [16363.352419] LNet: Removed LNI 192.168.202.126@tcp28 [16363.362481] LNet: Removed LNI 192.168.202.126@tcp29 [16363.374679] LNet: Removed LNI 192.168.202.126@tcp30 [16363.384393] LNet: Removed LNI 192.168.202.126@tcp31 [16363.391991] LNet: Removed LNI 192.168.202.126@tcp32 [16363.394784] LNet: Removed LNI 192.168.202.126@tcp33 [16363.399177] LNet: Removed LNI 192.168.202.126@tcp34 [16363.404904] LNet: Removed LNI 192.168.202.126@tcp35 [16363.411944] LNet: Removed LNI 192.168.202.126@tcp36 [16363.417756] LNet: Removed LNI 192.168.202.126@tcp37 [16363.429804] LNet: Removed LNI 192.168.202.126@tcp38 [16363.431518] LNet: Removed LNI 192.168.202.126@tcp39 [16363.441807] LNet: Removed LNI 192.168.202.126@tcp40 [16363.444699] LNet: Removed LNI 192.168.202.126@tcp41 [16363.449640] LNet: Removed LNI 192.168.202.126@tcp42 [16363.458599] LNet: Removed LNI 192.168.202.126@tcp43 [16363.464389] LNet: Removed LNI 192.168.202.126@tcp44 [16363.466983] LNet: Removed LNI 192.168.202.126@tcp45 [16363.471291] LNet: Removed LNI 192.168.202.126@tcp46 [16363.475447] LNet: Removed LNI 192.168.202.126@tcp47 [16363.485503] LNet: Removed LNI 192.168.202.126@tcp48 [16363.490898] LNet: Removed LNI 192.168.202.126@tcp49 [16363.520514] LNet: Removed LNI 192.168.202.126@tcp50 [16364.613364] Key type .llcrypt unregistered [16364.614882] Key type ._llcrypt unregistered [16389.228846] Key type ._llcrypt registered [16389.233200] Key type .llcrypt registered [16389.404087] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [16406.899700] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [16408.173950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16408.223411] alg: No test for adler32 (adler32-zlib) [16409.865302] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [16410.323703] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [16412.080320] Key type lgssc registered [16413.420820] Lustre: Echo OBD driver; http://www.lustre.org/ [16458.025379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16458.094913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16459.402505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16459.430176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16459.498943] Lustre: lustre-MDT0000: new disk, initializing [16459.571110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16459.586740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16464.587089] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [16476.031617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16476.175823] Lustre: 505707:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16476.324401] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16476.330025] Lustre: Skipped 1 previous similar message [16476.463144] Lustre: lustre-MDT0001: new disk, initializing [16476.579171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16476.614595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16476.628246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16481.277334] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [16490.953310] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16497.666578] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16506.079889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16506.368371] Lustre: *** cfs_fail_loc=90f, val=0*** [16506.504903] Lustre: lustre-OST0000: new disk, initializing [16506.508166] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16506.514557] Lustre: 507725:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16506.619568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16513.177590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16513.189706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16513.400215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16514.570897] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [16525.648733] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16539.110531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16539.120188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16539.133425] LustreError: Skipped 1 previous similar message [16539.138505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16539.171904] Lustre: Skipped 1 previous similar message [16539.197741] Lustre: Skipped 1 previous similar message [16541.770523] Lustre: server umount lustre-OST0000 complete [16548.327627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16548.344480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16548.364336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16549.364151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16549.396562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16549.411390] Lustre: Skipped 1 previous similar message [16553.062603] Lustre: server umount lustre-MDT0000 complete [16554.469256] LustreError: 505719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16554.507817] LustreError: 505719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [16557.911139] LustreError: 507724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788546340 with bad export cookie 18147789136146267542 [16557.911918] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16557.923511] LustreError: 507724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16558.431076] Lustre: server umount lustre-MDT0001 complete [16569.362676] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [16572.765401] Key type lgssc unregistered [16573.090924] LNet: 509635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16573.102452] LNetError: 509635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16573.114558] LNet: Removed LNI 192.168.202.126@tcp [16574.268492] Key type .llcrypt unregistered [16574.271303] Key type ._llcrypt unregistered [16598.129618] Key type ._llcrypt registered [16598.132807] Key type .llcrypt registered [16598.271141] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [16599.479098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16599.741585] alg: No test for adler32 (adler32-zlib) [16600.926517] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [16601.256750] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [16603.040251] Key type lgssc registered [16604.402342] Lustre: Echo OBD driver; http://www.lustre.org/ [16613.066554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16613.077120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16614.530390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16619.444366] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [16625.128524] LustreError: 510903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16625.141243] LustreError: 510903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16628.898068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16629.340593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16634.177215] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [16641.416240] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16647.204132] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16654.227513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16654.721054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16661.945594] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [16670.636461] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16680.968307] loop0: detected capacity change from 0 to 204800000 [16681.109367] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16681.131668] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16681.158313] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16688.358409] loop0: detected capacity change from 0 to 204800000 [16688.562805] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16688.575375] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16688.590823] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16694.723897] loop0: detected capacity change from 0 to 204800000 [16701.850700] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16702.039514] Lustre: fs15246-OST0000: new disk, initializing [16702.044749] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16702.052382] Lustre: 514496:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16702.133719] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16709.319876] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [16720.152190] loop1: detected capacity change from 0 to 204800000 [16725.875399] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16725.952696] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16726.071602] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16726.245700] Lustre: fs15246-MDT0000: new disk, initializing [16726.546491] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16726.591400] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16726.728577] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16727.788120] Lustre: 515619:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000271655db x1875426813199488/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788546526 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16731.389072] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [16732.905534] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16732.920312] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16732.960879] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16757.734250] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16757.770533] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16760.333845] Lustre: server umount fs15246-MDT0000 complete [16762.856257] LustreError: 516697:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16767.992878] LustreError: 516697:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16770.249393] Lustre: server umount fs15246-OST0000 complete [16782.821383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16782.833354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16782.861361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16788.210663] Lustre: server umount lustre-OST0000 complete [16793.059446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16793.069753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16793.089897] Lustre: Skipped 1 previous similar message [16793.097817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16793.105041] Lustre: Skipped 2 previous similar messages [16798.714388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16798.720906] Lustre: Skipped 2 previous similar messages [16803.814755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16807.394374] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16807.862715] Lustre: server umount lustre-MDT0000 complete [16809.958278] LustreError: 510904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16809.982776] LustreError: 510904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16812.840155] LustreError: 510883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788546595 with bad export cookie 14510691562706453196 [16812.844803] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16812.848351] LustreError: 510883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16813.511214] Lustre: server umount lustre-MDT0001 complete [16825.731279] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [16828.846531] Key type lgssc unregistered [16829.209932] LNet: 518718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16829.217994] LNetError: 518718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16829.244172] LNet: Removed LNI 192.168.202.126@tcp [16830.357103] Key type .llcrypt unregistered [16830.361958] Key type ._llcrypt unregistered [16852.958789] Key type ._llcrypt registered [16852.961558] Key type .llcrypt registered [16853.176820] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [16855.439134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16855.679047] alg: No test for adler32 (adler32-zlib) [16856.890861] Lustre: Lustre: Build Version: 2.17.58_5_ge0db421 [16857.137754] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [16858.928193] Key type lgssc registered [16860.295342] Lustre: Echo OBD driver; http://www.lustre.org/ [16902.579081] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [16915.474732] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [16955.243634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16955.337630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16956.674930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16956.712780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16956.861351] Lustre: lustre-MDT0000: new disk, initializing [16957.037738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16957.054890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16962.175156] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [16973.809106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16973.900317] Lustre: 523186:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16973.958817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16973.960785] Lustre: Skipped 1 previous similar message [16974.042621] Lustre: lustre-MDT0001: new disk, initializing [16974.139490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16974.167923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16974.175072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16978.955871] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [16988.254878] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16993.750052] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17000.559921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17000.768220] Lustre: lustre-OST0000: new disk, initializing [17000.773645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17000.780741] Lustre: 525154:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17000.901424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17006.725403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17006.739671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17006.909678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17007.252412] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [17019.300292] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17025.035173] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17025.264474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17030.588820] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17030.924098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17037.795942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17037.803695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17037.841766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17037.843949] LustreError: Skipped 1 previous similar message [17039.637807] Lustre: server umount lustre-OST0000 complete [17045.988883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17046.002995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17046.012529] Lustre: Skipped 1 previous similar message [17046.018094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17046.028128] Lustre: Skipped 1 previous similar message [17048.033231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17048.040661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17049.742206] Lustre: server umount lustre-MDT0000 complete [17053.041684] LustreError: 523178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788546835 with bad export cookie 947430956615089352 [17053.046705] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17053.054849] LustreError: 523178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17053.305475] Lustre: server umount lustre-MDT0001 complete [17081.812882] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [17092.509786] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [17130.253433] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16776 sec ========= 14:35:11 (1788546911) [17132.465577] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:35:13 (1788546913) === [17136.622373] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:35:17 (1788546917) === [17180.306257] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [17183.455097] Key type lgssc unregistered [17183.911408] LNet: 531800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17183.915822] LNetError: 531800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17184.934767] LNet: Removed LNI 192.168.202.126@tcp [17186.140522] Key type .llcrypt unregistered [17186.143641] Key type ._llcrypt unregistered