[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 482502555 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002004] Switched APIC routing to physical x2apic. [ 0.003010] kvm-guest: setup PV IPIs [ 0.005703] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007006] pid_max: default: 32768 minimum: 301 [ 0.009095] LSM: Security Framework initializing [ 0.010034] Yama: becoming mindful. [ 0.011029] SELinux: Initializing. [ 0.012067] *** VALIDATE selinux *** [ 0.019666] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023453] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024117] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026147] *** VALIDATE tmpfs *** [ 0.027477] *** VALIDATE proc *** [ 0.029096] *** VALIDATE cgroup *** [ 0.030000] *** VALIDATE cgroup2 *** [ 0.030318] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032016] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033030] Spectre V2 : User space: Vulnerable [ 0.034009] Speculative Store Bypass: Vulnerable [ 0.038060] debug: unmapping init [mem 0xffffffff85e59000-0xffffffff85e60fff] [ 0.040164] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041717] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042032] ... version: 2 [ 0.043007] ... bit width: 48 [ 0.044007] ... generic registers: 4 [ 0.045006] ... value mask: 0000ffffffffffff [ 0.046006] ... max period: 00007fffffffffff [ 0.047008] ... fixed-purpose events: 3 [ 0.048007] ... event mask: 000000070000000f [ 0.049300] rcu: Hierarchical SRCU implementation. [ 0.051553] smp: Bringing up secondary CPUs ... [ 0.052660] x86: Booting SMP configuration: [ 0.053015] .... node #0, CPUs: #1 #2 #3 [ 0.057016] smp: Brought up 1 node, 4 CPUs [ 0.059014] smpboot: Max logical packages: 1 [ 0.060010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.211284] node 0 deferred pages initialised in 149ms [ 0.213408] devtmpfs: initialized [ 0.214146] x86/mm: Memory block size: 128MB [ 0.216322] gcov: version magic: 0x41383552 [ 0.217522] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.220143] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.222392] pinctrl core: initialized pinctrl subsystem [ 0.227238] [ 0.228008] ************************************************************* [ 0.231036] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.236030] ** ** [ 0.240023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.242037] ** ** [ 0.247025] ** This means that this kernel is built to expose internal ** [ 0.250021] ** IOMMU data structures, which may compromise security on ** [ 0.255036] ** your system. ** [ 0.259019] ** ** [ 0.261011] ** If you see this message and you are not debugging the ** [ 0.264013] ** kernel, report this immediately to your vendor! ** [ 0.269029] ** ** [ 0.272013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.276016] ************************************************************* [ 0.280028] NET: Registered protocol family 16 [ 0.282712] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.283071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.286067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.289090] cpuidle: using governor menu [ 0.290000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.296110] PCI: Using configuration type 1 for base access [ 0.300276] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.318310] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.319063] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.321028] cryptd: max_cpu_qlen set to 1000 [ 0.324258] ACPI: Added _OSI(Module Device) [ 0.325027] ACPI: Added _OSI(Processor Device) [ 0.326012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.327052] ACPI: Added _OSI(Processor Aggregator Device) [ 0.331728] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.340770] ACPI: Interpreter enabled [ 0.343064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.345046] ACPI: Using IOAPIC for interrupt routing [ 0.348093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.353760] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.373577] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.377036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.381017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.384072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.388320] acpiphp: Slot [2] registered [ 0.389084] acpiphp: Slot [5] registered [ 0.390082] acpiphp: Slot [6] registered [ 0.391077] acpiphp: Slot [7] registered [ 0.393090] acpiphp: Slot [8] registered [ 0.394000] acpiphp: Slot [9] registered [ 0.394000] acpiphp: Slot [10] registered [ 0.399125] acpiphp: Slot [3] registered [ 0.401100] acpiphp: Slot [4] registered [ 0.404127] acpiphp: Slot [11] registered [ 0.407043] acpiphp: Slot [12] registered [ 0.408064] acpiphp: Slot [13] registered [ 0.409067] acpiphp: Slot [14] registered [ 0.410062] acpiphp: Slot [15] registered [ 0.411092] acpiphp: Slot [16] registered [ 0.412066] acpiphp: Slot [17] registered [ 0.413077] acpiphp: Slot [18] registered [ 0.414064] acpiphp: Slot [19] registered [ 0.416062] acpiphp: Slot [20] registered [ 0.417088] acpiphp: Slot [21] registered [ 0.418081] acpiphp: Slot [22] registered [ 0.419058] acpiphp: Slot [23] registered [ 0.420065] acpiphp: Slot [24] registered [ 0.421059] acpiphp: Slot [25] registered [ 0.422058] acpiphp: Slot [26] registered [ 0.423061] acpiphp: Slot [27] registered [ 0.425072] acpiphp: Slot [28] registered [ 0.426084] acpiphp: Slot [29] registered [ 0.428063] acpiphp: Slot [30] registered [ 0.429059] acpiphp: Slot [31] registered [ 0.430074] PCI host bridge to bus 0000:00 [ 0.431012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.433017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.434011] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.436022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.438016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.441016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.442141] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.444000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.445828] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.459764] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.468058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.474018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.479017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.481011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.482600] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.484711] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.486030] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.488622] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.491931] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.501013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.505010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.512409] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.522017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.534016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.546018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.555271] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.561018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.565017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.584016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.599149] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.613024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.621030] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.649011] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.658000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.664015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.670101] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.683020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.695000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.699015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.706032] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.720019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.732932] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.743023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.751017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.768015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.784207] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.786377] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.789192] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.792962] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.796193] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.802054] iommu: Default domain type: Passthrough [ 0.803343] SCSI subsystem initialized [ 0.804128] ACPI: bus type USB registered [ 0.806093] usbcore: registered new interface driver usbfs [ 0.808205] usbcore: registered new interface driver hub [ 0.811208] usbcore: registered new device driver usb [ 0.813179] pps_core: LinuxPPS API ver. 1 registered [ 0.815066] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.818065] PTP clock support registered [ 0.820139] EDAC MC: Ver: 3.0.0 [ 0.821160] PCI: Using ACPI for IRQ routing [ 0.822923] NetLabel: Initializing [ 0.825008] NetLabel: domain hash size = 128 [ 0.826006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.827064] NetLabel: unlabeled traffic allowed by default [ 0.829156] vgaarb: loaded [ 0.830228] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.831011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.835408] clocksource: Switched to clocksource kvm-clock [ 0.939720] VFS: Disk quotas dquot_6.6.0 [ 0.942076] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.944584] *** VALIDATE ramfs *** [ 0.945687] *** VALIDATE hugetlbfs *** [ 0.947491] pnp: PnP ACPI init [ 0.950102] pnp: PnP ACPI: found 6 devices [ 0.966592] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.969018] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.971077] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.972243] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.973566] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.974836] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.976420] NET: Registered protocol family 2 [ 0.977936] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.981281] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.985038] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.989361] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.992067] TCP: Hash tables configured (established 65536 bind 65536) [ 0.994414] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.996945] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.999479] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.001960] NET: Registered protocol family 1 [ 1.004199] RPC: Registered named UNIX socket transport module. [ 1.005767] RPC: Registered udp transport module. [ 1.007284] RPC: Registered tcp transport module. [ 1.008705] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.011237] NET: Registered protocol family 44 [ 1.012830] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.014958] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.017058] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.019177] PCI: CLS 0 bytes, default 64 [ 1.020754] Unpacking initramfs... [ 2.519223] debug: unmapping init [mem 0xffff89547cc54000-0xffff89547ffbffff] [ 2.524285] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.527756] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.534216] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.156550] Initialise system trusted keyrings [ 3.160166] Key type blacklist registered [ 3.163351] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.178681] zbud: loaded [ 3.184671] *** VALIDATE nfs *** [ 3.185693] *** VALIDATE nfs4 *** [ 3.187013] pstore: using deflate compression [ 3.190921] Platform Keyring initialized [ 3.294544] NET: Registered protocol family 38 [ 3.295936] Key type asymmetric registered [ 3.297408] Asymmetric key parser 'x509' registered [ 3.299403] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.301965] io scheduler mq-deadline registered [ 3.303075] io scheduler kyber registered [ 3.306636] io scheduler bfq registered [ 3.308279] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.310872] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.313549] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.315992] ACPI: Power Button [PWRF] [ 3.321996] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.329812] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.342895] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.353458] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.367897] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.399537] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.428361] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.432255] Non-volatile memory driver v1.3 [ 3.433439] Linux agpgart interface v0.103 [ 3.462538] virtio_blk virtio1: [vda] 134712 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.465294] vda: detected capacity change from 0 to 68972544 [ 3.478468] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.480813] vdb: detected capacity change from 0 to 1073741824 [ 3.494156] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.496275] vdc: detected capacity change from 0 to 2621440000 [ 3.508684] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.510878] vdd: detected capacity change from 0 to 2621440000 [ 3.523556] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.525850] vde: detected capacity change from 0 to 4294967296 [ 3.541804] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.544214] vdf: detected capacity change from 0 to 4294967296 [ 3.549811] libphy: Fixed MDIO Bus: probed [ 3.556840] usbcore: registered new interface driver usbserial_generic [ 3.559215] usbserial: USB Serial support registered for generic [ 3.561423] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.566461] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.568192] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.570514] mousedev: PS/2 mouse device common for all mice [ 3.572711] rtc_cmos 00:05: RTC can wake from S4 [ 3.580613] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.584492] rtc_cmos 00:05: registered as rtc0 [ 3.586556] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.588458] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.591988] intel_pstate: CPU model not supported [ 3.596703] hid: raw HID events driver (C) Jiri Kosina [ 3.599097] usbcore: registered new interface driver usbhid [ 3.600713] usbhid: USB HID core driver [ 3.602809] drop_monitor: Initializing network drop monitor service [ 3.604472] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.605175] Initializing XFRM netlink socket [ 3.606082] NET: Registered protocol family 10 [ 3.612704] Segment Routing with IPv6 [ 3.613923] NET: Registered protocol family 17 [ 3.615459] mpls_gso: MPLS GSO support [ 3.621070] RAS: Correctable Errors collector initialized. [ 3.622719] AVX version of gcm_enc/dec engaged. [ 3.624010] AES CTR mode by8 optimization enabled [ 3.704977] sched_clock: Marking stable (3704961937, 0)->(4550794320, -845832383) [ 3.708498] registered taskstats version 1 [ 3.710520] Loading compiled-in X.509 certificates [ 3.712280] zswap: loaded using pool lzo/zbud [ 3.741438] Key type big_key registered [ 3.755940] Key type encrypted registered [ 3.757531] ima: No TPM chip found, activating TPM-bypass! [ 3.758954] ima: Allocated hash algorithm: sha1 [ 3.760555] ima: No architecture policies found [ 3.762704] evm: Initialising EVM extended attributes: [ 3.764218] evm: security.selinux [ 3.765228] evm: security.ima [ 3.766382] evm: security.capability [ 3.767454] evm: HMAC attrs: 0x1 [ 3.769450] rtc_cmos 00:05: setting system clock to 2026-03-06 00:42:05 UTC (1772757725) [ 3.775109] debug: unmapping init [mem 0xffffffff86e03000-0xffffffff86ffffff] [ 3.777742] debug: unmapping init [mem 0xffffffff85b82000-0xffffffff85e58fff] [ 3.785092] Write protecting the kernel read-only data: 28672k [ 3.788129] debug: unmapping init [mem 0xffffffff84203000-0xffffffff843fffff] [ 3.790468] debug: unmapping init [mem 0xffffffff84b14000-0xffffffff84bfffff] [ 3.823502] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.830984] systemd[1]: Detected virtualization kvm. [ 3.832643] systemd[1]: Detected architecture x86-64. [ 3.834389] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.861620] systemd[1]: No hostname configured. [ 3.863274] systemd[1]: Set hostname to . [ 3.865102] random: systemd: uninitialized urandom read (16 bytes read) [ 3.867438] systemd[1]: Initializing machine ID from random generator. [ 3.954448] random: ln: uninitialized urandom read (6 bytes read) [ 4.062827] random: systemd: uninitialized urandom read (16 bytes read) [ 4.066420] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.073966] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.079828] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.952386] device-mapper: uevent: version 1.0.3 [ 4.954546] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 5.917871] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.105073] random: fast init done [ 6.145525] scsi host0: ata_piix [ 6.250956] scsi host1: ata_piix [ 6.254664] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.262410] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.279289] random: crng init done [ 11.280737] random: 7 urandom warning(s) missed due to ratelimiting [ 13.876913] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.715341] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.220327] printk: systemd: 25 output lines suppressed due to ratelimiting [ 16.809487] SELinux: Disabled at runtime. [ 16.903852] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.909520] systemd[1]: Detected virtualization kvm. [ 16.910758] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.037634] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.045769] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.067599] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.074251] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.083072] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.097515] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.107874] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ 18.176563] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 18.888984] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.908848] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.136164] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.879308] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.013182] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit)[ 25.154786] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit)[ 25.503310] NFS: Registering the id_resolver key type [ 25.505208] Key type id_resolver registered [ 25.506693] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg319-server login: [ 51.346269] libcfs: loading out-of-tree module taints kernel. [ 51.363114] Key type ._llcrypt registered [ 51.364638] Key type .llcrypt registered [ 51.400824] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 58.288261] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 58.820656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 58.826968] alg: No test for adler32 (adler32-zlib) [ 59.790084] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 60.058745] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 61.663160] Key type lgssc registered [ 62.140702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 68.108403] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 81.701672] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 86.261967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 86.270848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 87.373731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 87.385393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 87.428630] Lustre: lustre-MDT0000: new disk, initializing [ 87.452788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 87.460932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 88.811893] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 93.823602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 93.860689] Lustre: 6485:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 93.874083] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 93.876191] Lustre: Skipped 1 previous similar message [ 93.917741] Lustre: lustre-MDT0001: new disk, initializing [ 93.938810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 93.946886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 93.950437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 95.274922] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 97.517732] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 100.631704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 100.718415] Lustre: lustre-OST0000: new disk, initializing [ 100.721223] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 100.739653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 102.591537] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 104.490603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 104.493183] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 104.513444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 107.652550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 107.693042] Lustre: lustre-OST0001: new disk, initializing [ 107.694981] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 107.710245] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 109.458360] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.529166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 111.531435] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 111.544455] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 115.065128] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 121.818682] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 128.592625] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing check_logdir /tmp/testlogs/ [ 130.168631] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing yml_node [ 132.048964] Lustre: DEBUG MARKER: Client: 2.17.50.193 [ 133.120055] Lustre: DEBUG MARKER: MDS: 2.17.50.193 [ 134.034445] Lustre: DEBUG MARKER: OSS: 2.17.50.193 [ 134.599299] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Mar 5 19:44:15 EST 2026 [ 140.540205] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 141.035656] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 155.615476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 155.618991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 155.623422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 157.663806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 157.664140] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 157.666599] Lustre: Skipped 2 previous similar messages [ 157.672239] Lustre: Skipped 2 previous similar messages [ 158.875218] Lustre: server umount lustre-MDT0000 complete [ 160.079354] LustreError: 6477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772757881 with bad export cookie 7328019918456856293 [ 160.079753] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 160.082841] LustreError: 6477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 162.783763] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 162.783965] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 162.789110] Lustre: Skipped 1 previous similar message [ 165.792237] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 165.794408] Lustre: Skipped 2 previous similar messages [ 166.237660] Lustre: server umount lustre-MDT0001 complete [ 167.500292] Lustre: server umount lustre-OST0000 complete [ 168.876892] Lustre: server umount lustre-OST0001 complete [ 171.012989] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 174.027529] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 186.422656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.517928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 186.534409] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 186.573237] Lustre: lustre-MDT0000: new disk, initializing [ 186.607879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.617345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 188.002364] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 192.547197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.598594] Lustre: 14944:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 192.718820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 194.031688] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 197.250711] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 198.561332] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 201.030172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.143072] Lustre: lustre-OST0000: new disk, initializing [ 201.145139] Lustre: Skipped 1 previous similar message [ 201.147800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 201.151318] Lustre: Skipped 2 previous similar messages [ 201.186394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.189502] Lustre: Skipped 1 previous similar message [ 202.559297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 202.562865] Lustre: Skipped 1 previous similar message [ 202.565385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 202.604981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 203.382298] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 206.799718] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 208.369545] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 208.461679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 209.966723] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 210.056394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 212.963574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 212.964144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 212.965061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 212.968676] Lustre: Skipped 1 previous similar message [ 212.975540] LustreError: Skipped 1 previous similar message [ 223.203440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 223.205544] Lustre: Skipped 4 previous similar messages [ 225.247109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 225.302460] Lustre: server umount lustre-OST0000 complete [ 228.322332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 228.328540] Lustre: Skipped 1 previous similar message [ 241.119108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 241.178882] Lustre: server umount lustre-MDT0000 complete [ 242.416199] LustreError: 16684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772757964 with bad export cookie 7328019918456857812 [ 242.419612] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 242.420658] LustreError: 16684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 242.534685] Lustre: server umount lustre-MDT0001 complete [ 243.890832] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:46:05 (1772757965) [ 244.643943] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 245.226181] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:46:06 (1772757966) [ 247.449820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 247.596974] LustreError: 18948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 247.603376] LustreError: 18948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 247.620456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 248.886206] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 251.591815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 252.920852] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 254.657126] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 255.850285] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 258.190379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.259136] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 262.095866] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 282.080064] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 282.086420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 282.091692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 282.094331] Lustre: Skipped 6 previous similar messages [ 286.557595] Lustre: server umount lustre-OST0000 complete [ 294.106951] Lustre: server umount lustre-MDT0000 complete [ 295.505224] LustreError: 18928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772758017 with bad export cookie 7328019918456858694 [ 295.507989] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 295.510863] LustreError: 18928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 295.643395] Lustre: server umount lustre-MDT0001 complete [ 299.669312] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 300.916241] Key type lgssc unregistered [ 301.055446] LNet: 22147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.063496] LNetError: 22147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 301.076443] LNet: Removed LNI 192.168.203.119@tcp [ 301.473165] Key type .llcrypt unregistered [ 301.474556] Key type ._llcrypt unregistered [ 311.965577] Key type ._llcrypt registered [ 311.967141] Key type .llcrypt registered [ 312.014616] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 312.442736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 312.455286] alg: No test for adler32 (adler32-zlib) [ 313.331568] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 313.436092] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 315.039209] Key type lgssc registered [ 315.523194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 318.903295] loop: module loaded [ 318.927577] loop0: detected capacity change from 0 to 204800000 [ 319.070131] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 319.074285] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 319.225629] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 321.538184] loop0: detected capacity change from 0 to 204800000 [ 321.652268] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 321.656271] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 321.660868] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 323.489719] loop0: detected capacity change from 0 to 204800000 [ 325.543196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 325.549826] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 326.650335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 326.671519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 326.708614] Lustre: lustre-MDT0000: new disk, initializing [ 326.733790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 326.741243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 328.135634] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 331.870721] loop1: detected capacity change from 0 to 204800000 [ 333.953281] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 334.055676] Lustre: lustre-OST07c6: new disk, initializing [ 334.057597] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 334.059491] Lustre: Skipped 1 previous similar message [ 334.083412] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 334.236273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 334.239562] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 334.251387] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 336.020467] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 338.213299] Lustre: 25285:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 338.216389] Lustre: 25285:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 338.218928] Lustre: 25285:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 338.223031] Lustre: 25285:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 338.225885] Lustre: 25285:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 338.228557] Lustre: 25285:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 356.816246] Lustre: 24279:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 356.819807] Lustre: 24279:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 356.822779] Lustre: 24279:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 356.825600] Lustre: 24279:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 356.829286] Lustre: 24279:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 356.831616] Lustre: 24279:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 360.355048] Lustre: 24280:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 360.358294] Lustre: 24280:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 360.361262] Lustre: 24280:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 360.364299] Lustre: 24280:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 360.366531] Lustre: 24280:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 360.368693] Lustre: 24280:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 360.371302] Lustre: 24280:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 360.374149] Lustre: 24280:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 360.376712] Lustre: 24280:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 360.379720] Lustre: 24280:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 360.382987] Lustre: 24280:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 360.385352] Lustre: 24280:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 363.445560] Lustre: 24279:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 363.448270] Lustre: 24279:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 363.450315] Lustre: 24279:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 363.452455] Lustre: 24279:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 363.454423] Lustre: 24279:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 363.456462] Lustre: 24279:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 363.458541] Lustre: 24279:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 363.461031] Lustre: 24279:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 363.463058] Lustre: 24279:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 363.465037] Lustre: 24279:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 363.467032] Lustre: 24279:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 363.468669] Lustre: 24279:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 370.001840] Lustre: 24278:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 370.004119] Lustre: 24278:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 3 previous similar messages [ 370.006011] Lustre: 24278:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 370.007834] Lustre: 24278:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 370.010052] Lustre: 24278:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 370.013350] Lustre: 24278:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 370.016784] Lustre: 24278:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 370.019757] Lustre: 24278:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 370.023026] Lustre: 24278:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 370.025585] Lustre: 24278:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 370.027575] Lustre: 24278:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 370.029569] Lustre: 24278:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 379.581874] Lustre: 24278:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 379.584241] Lustre: 24278:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 5 previous similar messages [ 379.586285] Lustre: 24278:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 379.588260] Lustre: 24278:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 379.590648] Lustre: 24278:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 379.593888] Lustre: 24278:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 379.597392] Lustre: 24278:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 379.600159] Lustre: 24278:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 379.602141] Lustre: 24278:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 379.603717] Lustre: 24278:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 379.606219] Lustre: 24278:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 379.608009] Lustre: 24278:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 421.344964] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 421.344975] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 421.350457] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 422.357285] Lustre: server umount lustre-OST07c6 complete [ 425.666675] Lustre: server umount lustre-MDT0000 complete [ 430.873981] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 431.900425] Key type lgssc unregistered [ 432.036334] LNet: 27796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.039227] LNetError: 27796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 432.051307] LNet: Removed LNI 192.168.203.119@tcp [ 432.350719] Key type .llcrypt unregistered [ 432.352345] Key type ._llcrypt unregistered [ 439.595235] Key type ._llcrypt registered [ 439.596565] Key type .llcrypt registered [ 439.631659] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 439.966479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 440.002609] alg: No test for adler32 (adler32-zlib) [ 440.868675] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 440.958586] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 442.543154] Key type lgssc registered [ 442.901288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 445.471796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 445.478286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.639044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.998226] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 450.851317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.973910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 452.362923] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 454.136073] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 455.338488] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 457.512780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 457.625084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.434274] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 461.206893] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 471.970657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 482.273119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.274470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 482.275026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 482.277893] Lustre: Skipped 1 previous similar message [ 487.376642] Lustre: server umount lustre-OST0000 complete [ 491.999881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 492.002555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.006423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 492.008211] Lustre: Skipped 1 previous similar message [ 494.800888] Lustre: server umount lustre-MDT0000 complete [ 495.998993] LustreError: 29008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772758217 with bad export cookie 15507912222460815063 [ 496.001650] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 496.002322] LustreError: 29008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 496.115875] Lustre: server umount lustre-MDT0001 complete [ 499.608446] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 500.570934] Key type lgssc unregistered [ 500.685737] LNet: 32425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.688654] LNetError: 32425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 500.696616] LNet: Removed LNI 192.168.203.119@tcp [ 500.982749] Key type .llcrypt unregistered [ 500.984140] Key type ._llcrypt unregistered [ 508.040530] Key type ._llcrypt registered [ 508.042092] Key type .llcrypt registered [ 508.076644] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 508.377022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 508.384466] alg: No test for adler32 (adler32-zlib) [ 509.225667] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 509.305320] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 510.879130] Key type lgssc registered [ 511.210266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 513.649682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 513.655215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 514.800965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.082451] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 518.800277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.915120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 520.147929] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 521.868992] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 523.061150] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 525.307955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 525.421341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.249995] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 528.991120] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 540.774319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 549.344030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 549.346144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.350195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 551.394061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 551.394603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.394643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 554.509869] Lustre: server umount lustre-OST0000 complete [ 556.512520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.517542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 556.519659] Lustre: Skipped 2 previous similar messages [ 561.632056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 561.634433] Lustre: Skipped 1 previous similar message [ 566.752052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 566.754361] Lustre: Skipped 1 previous similar message [ 570.335113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 570.401703] Lustre: server umount lustre-MDT0000 complete [ 571.765621] LustreError: 35820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772758293 with bad export cookie 1820572463321414530 [ 571.767971] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 571.770165] LustreError: 35820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 571.897337] Lustre: server umount lustre-MDT0001 complete [ 575.444769] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 576.422794] Key type lgssc unregistered [ 576.553378] LNet: 36856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 576.555810] LNetError: 36856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 576.564260] LNet: Removed LNI 192.168.203.119@tcp [ 576.838459] Key type .llcrypt unregistered [ 576.839643] Key type ._llcrypt unregistered [ 585.831836] Key type ._llcrypt registered [ 585.833192] Key type .llcrypt registered [ 585.879480] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 586.287350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 586.364596] alg: No test for adler32 (adler32-zlib) [ 587.242559] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 587.326964] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 588.911163] Key type lgssc registered [ 589.303356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 591.832464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 591.838752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 592.998736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.250939] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 597.050656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 597.177134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 598.448339] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 600.241729] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 601.393966] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 603.647696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 603.753924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 605.537477] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 607.357539] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 618.086738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 628.191744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 628.193965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.197862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.705296] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.711497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.714093] Lustre: Skipped 2 previous similar messages [ 631.634318] Lustre: server umount lustre-MDT0000 complete [ 632.811090] LustreError: 39856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772758354 with bad export cookie 9748772359933524267 [ 632.814447] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 632.815133] LustreError: 39856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 633.395283] LustreError: 38951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.400318] LustreError: 38951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 633.402332] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.19@tcp (stopping) [ 633.824298] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.828093] Lustre: Skipped 1 previous similar message [ 637.855771] LustreError: 40396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.855963] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 637.861273] LustreError: 40396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 637.865915] Lustre: Skipped 1 previous similar message [ 642.976245] LustreError: 38236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.976438] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 642.981422] LustreError: 38236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 642.983656] Lustre: Skipped 1 previous similar message [ 647.135106] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 647.212075] Lustre: server umount lustre-MDT0001 complete [ 673.249094] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 673.315111] Lustre: server umount lustre-OST0000 complete [ 678.620689] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 679.590413] Key type lgssc unregistered [ 679.713265] LNet: 41771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.715431] LNetError: 41771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.724252] LNet: Removed LNI 192.168.203.119@tcp [ 679.999540] Key type .llcrypt unregistered [ 680.000721] Key type ._llcrypt unregistered [ 688.243118] Key type ._llcrypt registered [ 688.244172] Key type .llcrypt registered [ 688.278768] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 688.693963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 688.700781] alg: No test for adler32 (adler32-zlib) [ 689.536305] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 689.609368] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 691.183161] Key type lgssc registered [ 691.516332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 693.974774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 693.981562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.134128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.334195] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 698.985485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.096679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 700.333696] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 701.991344] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 703.099239] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 705.068455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 705.169824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 706.213416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 706.955721] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 708.728885] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 714.721989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 714.724997] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 714.726892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 714.730608] Lustre: Skipped 1 previous similar message [ 715.323260] Lustre: lustre-OST0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 717.895931] Lustre: server umount lustre-OST0000 complete [ 734.497144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 734.497849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 734.500535] Lustre: Skipped 2 previous similar messages [ 734.503071] Lustre: Skipped 1 previous similar message [ 736.660844] Lustre: server umount lustre-MDT0000 complete [ 737.808095] LustreError: 45393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772758459 with bad export cookie 11767912710512825571 [ 737.810788] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 737.811041] LustreError: 45393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 752.095080] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 752.181801] Lustre: server umount lustre-MDT0001 complete [ 755.606767] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 756.493709] Key type lgssc unregistered [ 756.613283] LNet: 46506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 756.615474] LNetError: 46506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 756.624271] LNet: Removed LNI 192.168.203.119@tcp [ 756.884096] Key type .llcrypt unregistered [ 756.885087] Key type ._llcrypt unregistered [ 765.244794] Key type ._llcrypt registered [ 765.245733] Key type .llcrypt registered [ 765.286183] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 765.665566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 765.711460] alg: No test for adler32 (adler32-zlib) [ 766.566329] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 766.654208] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 768.239188] Key type lgssc registered [ 768.587927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 770.986957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 770.992567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.123496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 773.269383] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 775.849179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 775.965156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 777.161710] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 778.834368] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 779.970152] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 782.070588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 782.175903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 783.206173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 784.021219] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 785.621515] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 788.535734] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 789.569082] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 800.298396] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-0519h55m21s [ 800.792516] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 801.761899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 801.762069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 801.762775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 801.770558] Lustre: Skipped 2 previous similar messages [ 802.363700] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 802.366808] Lustre: Skipped 2 previous similar messages [ 803.379140] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 803.381416] Lustre: Skipped 3 previous similar messages [ 807.641784] Lustre: server umount lustre-MDT0000 complete [ 808.740854] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 808.928698] LustreError: 49337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 808.934132] LustreError: 49337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 809.522717] LustreError: 47876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 810.937159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.989134] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 811.088358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.116106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 812.134787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 812.461601] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 813.622077] Lustre: DEBUG MARKER: Wait for df (36016) ... [ 814.167151] Lustre: DEBUG MARKER: done [ 816.096289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 816.098643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 816.102501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 817.120088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 817.124030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 817.237138] Lustre: server umount lustre-OST0000 complete [ 822.752675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 822.752711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 822.757897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 822.761911] Lustre: Skipped 2 previous similar messages [ 824.721484] Lustre: server umount lustre-MDT0000 complete [ 825.848076] LustreError: 47863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772758547 with bad export cookie 10459834275893668620 [ 825.850872] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 825.850877] LustreError: 47863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 825.951127] Lustre: server umount lustre-MDT0001 complete [ 829.335378] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 830.310494] Key type lgssc unregistered [ 830.426364] LNet: 52427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.428721] LNetError: 52427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.435322] LNet: Removed LNI 192.168.203.119@tcp [ 830.729578] Key type .llcrypt unregistered [ 830.731180] Key type ._llcrypt unregistered [ 844.536899] Key type ._llcrypt registered [ 844.537988] Key type .llcrypt registered [ 844.571634] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 844.992937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 845.020348] alg: No test for adler32 (adler32-zlib) [ 845.857115] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 845.926924] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 847.503127] Key type lgssc registered [ 847.814756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 850.083811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 850.089361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.171345] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 851.177624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 851.232693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 852.415722] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 855.031423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.064581] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 855.073631] Lustre: 54595:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 855.161353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 856.334655] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 857.972265] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 859.016607] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 861.106288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 861.168163] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 861.225069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.937224] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 864.629067] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 866.331231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 872.999786] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-0519h56m34s [ 873.493815] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 875.686464] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 876.420738] Lustre: DEBUG MARKER: Creating a test file and stat it [ 876.432763] LustreError: 54601:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 896.576121] Lustre: lustre-MDT0000: Client fe98236e-fbc2-46aa-b40f-f15ab9695cd3 (at 192.168.203.19@tcp) reconnecting [ 916.479157] LustreError: 54601:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 917.152371] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 917.906647] Lustre: DEBUG MARKER: done [ 922.089989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 922.093574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.097908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 922.594140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 925.467654] Lustre: server umount lustre-OST0000 complete [ 927.199789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 927.202863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 927.207588] Lustre: Skipped 1 previous similar message [ 927.209714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 927.211571] Lustre: Skipped 1 previous similar message [ 928.224366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 928.226620] Lustre: Skipped 2 previous similar messages [ 932.824948] Lustre: server umount lustre-MDT0000 complete [ 934.067137] LustreError: 54587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772758655 with bad export cookie 3815268950318199098 [ 934.069555] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 934.069860] LustreError: 54587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 934.194416] Lustre: server umount lustre-MDT0001 complete [ 937.568869] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 938.527366] Key type lgssc unregistered [ 938.652356] LNet: 58561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.654892] LNetError: 58561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.662645] LNet: Removed LNI 192.168.203.119@tcp [ 938.958763] Key type .llcrypt unregistered [ 938.959987] Key type ._llcrypt unregistered [ 953.585885] Key type ._llcrypt registered [ 953.586858] Key type .llcrypt registered [ 953.616979] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 954.004553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 954.010929] alg: No test for adler32 (adler32-zlib) [ 954.858147] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 954.931088] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 956.503130] Key type lgssc registered [ 956.848596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 957.845297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 957.850512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.938860] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 958.946349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 959.007740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 959.311835] Lustre: Failing over lustre-MDT0000 [ 959.404776] Lustre: server umount lustre-MDT0000 complete [ 961.432294] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:58:02 (1772758682) [ 963.419449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.547873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.790729] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 967.344257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 967.374840] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 967.387020] Lustre: 61323:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 967.485831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 968.656361] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 970.307609] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 971.452139] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 973.532732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 973.599082] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 973.665904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.436816] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 976.991787] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 977.935678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 985.521373] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 988.129677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.129936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 988.130279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 988.133397] Lustre: Skipped 2 previous similar messages [ 993.239851] Lustre: server umount lustre-MDT0000 complete [ 994.397840] LustreError: 61313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772758716 with bad export cookie 12028900196831000503 [ 994.399856] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 994.401062] LustreError: 61313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 994.496717] Lustre: server umount lustre-MDT0001 complete [ 995.515670] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 996.927840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 999.533164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.511657] LustreError: 60244:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000987c7ce2 x1858872201476992/t0(0) o250->MGC192.168.203.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1004.619241] LustreError: 62942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.626487] LustreError: 62942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1004.654815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.685207] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1005.884626] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1008.545961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.646375] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.963374] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1011.586474] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1012.701644] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1013.732414] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1014.751261] Lustre: 60248:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772758720/real 1772758720] req@00000000be14451b x1858872201476608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772758736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1015.775142] Lustre: 60246:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772758721/real 1772758721] req@00000000b29be5cc x1858872201476736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772758737 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1018.853103] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1020.319171] Lustre: 60245:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772758725/real 1772758725] req@000000002241304a x1858872201476864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772758741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1023.970848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.970978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1023.971981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1023.971986] Lustre: Skipped 2 previous similar messages [ 1023.975491] Lustre: Skipped 2 previous similar messages [ 1025.435255] Lustre: server umount lustre-MDT0000 complete [ 1026.644859] LustreError: 61313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772758748 with bad export cookie 12028900196831002771 [ 1026.646544] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1026.647639] LustreError: 61313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1026.745974] Lustre: server umount lustre-MDT0001 complete [ 1028.763297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1031.602625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1037.215878] LustreError: 60244:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000798cebc7 x1858872201496448/t0(0) o250->MGC192.168.203.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1037.348050] LustreError: 62943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1037.374553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1037.376692] Lustre: Skipped 1 previous similar message [ 1037.392689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1038.692643] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1041.621408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1041.718870] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1043.093792] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1043.812749] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1044.832091] Lustre: 60245:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772758750/real 1772758750] req@0000000096252c8b x1858872201496064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772758766 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1044.840590] Lustre: 60245:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1044.852187] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1045.998443] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1049.056858] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.058833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1049.061519] Lustre: Skipped 1 previous similar message [ 1049.064886] Lustre: Skipped 4 previous similar messages [ 1050.079160] Lustre: 60247:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772758755/real 1772758755] req@0000000006fae2ca x1858872201496320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772758771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1050.088182] Lustre: 60247:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1053.849188] Lustre: server umount lustre-MDT0000 complete [ 1054.176606] LustreError: 67208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1054.182025] LustreError: 67208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1055.178168] LustreError: 62949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772758776 with bad export cookie 12028900196831004150 [ 1055.180971] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1055.182610] LustreError: 62949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1055.289484] Lustre: server umount lustre-MDT0001 complete [ 1057.406392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1058.538794] Lustre: DEBUG MARKER: files compared the same [ 1068.441331] Lustre: server umount lustre-OST0000 complete [ 1073.460833] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1074.382353] Key type lgssc unregistered [ 1074.505328] LNet: 70239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.507787] LNetError: 70239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.520328] LNet: Removed LNI 192.168.203.119@tcp [ 1074.814503] Key type .llcrypt unregistered [ 1074.815940] Key type ._llcrypt unregistered [ 1082.975345] Key type ._llcrypt registered [ 1082.976904] Key type .llcrypt registered [ 1083.011455] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1083.308892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1083.342768] alg: No test for adler32 (adler32-zlib) [ 1084.191955] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 1084.271260] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1085.847128] Key type lgssc registered [ 1086.179446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1088.578517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1088.584591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.725916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.938610] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug +malloc all [ 1093.525455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.616742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1094.813914] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug +malloc all [ 1096.444554] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1097.497460] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1099.488293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1099.571364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.579473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1101.283363] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug +malloc all [ 1102.863548] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1108.960283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.960665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1108.960713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1108.963481] Lustre: Skipped 1 previous similar message [ 1112.333628] Lustre: server umount lustre-OST0000 complete [ 1114.079952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.080062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1114.080218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.080224] Lustre: Skipped 1 previous similar message [ 1114.084129] Lustre: Skipped 1 previous similar message [ 1119.199637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1119.203453] Lustre: Skipped 2 previous similar messages [ 1119.766329] Lustre: server umount lustre-MDT0000 complete [ 1121.003625] LustreError: 71594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772758842 with bad export cookie 3587540455681543087 [ 1121.005071] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1121.007788] LustreError: 71594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1121.117606] Lustre: server umount lustre-MDT0001 complete [ 1124.578251] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1125.578974] Key type lgssc unregistered [ 1125.697412] LNet: 74788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.700790] LNetError: 74788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.708283] LNet: Removed LNI 192.168.203.119@tcp [ 1125.973810] Key type .llcrypt unregistered [ 1125.975081] Key type ._llcrypt unregistered [ 1134.896530] Key type ._llcrypt registered [ 1134.897602] Key type .llcrypt registered [ 1134.941092] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1135.495897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1135.538717] alg: No test for adler32 (adler32-zlib) [ 1136.410765] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 1136.505291] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1138.095237] Key type lgssc registered [ 1138.497424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1141.225931] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1141.232362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1156.639248] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1161.759264] LustreError: 76152:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.119@tcp: failed processing log, type 4: rc = -110 [ 1188.383231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1190.263519] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1191.953298] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1194.484789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.521037] Lustre: *** cfs_fail_loc=706, val=0*** [ 1194.648161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1194.677615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1195.908253] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1198.762857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.886606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1200.181384] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1202.027789] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1203.312492] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1204.192389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1204.196174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.200739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1205.215791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1205.218493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1205.222806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1208.290435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1208.293607] Lustre: Skipped 1 previous similar message [ 1210.120562] Lustre: server umount lustre-OST0000 complete [ 1213.408730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.409463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1213.413111] Lustre: Skipped 1 previous similar message [ 1217.492469] Lustre: server umount lustre-MDT0000 complete [ 1218.527990] LustreError: 78680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1218.532784] LustreError: 78680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1218.712265] LustreError: 76158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772758940 with bad export cookie 2153717084092939320 [ 1218.714926] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1218.715661] LustreError: 76158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1218.834349] Lustre: server umount lustre-MDT0001 complete [ 1222.342270] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1223.371274] Key type lgssc unregistered [ 1223.501444] LNet: 79413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.504317] LNetError: 79413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1223.513350] LNet: Removed LNI 192.168.203.119@tcp [ 1223.824677] Key type .llcrypt unregistered [ 1223.825866] Key type ._llcrypt unregistered [ 1233.083397] Key type ._llcrypt registered [ 1233.084383] Key type .llcrypt registered [ 1233.127434] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1233.483263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1233.490675] alg: No test for adler32 (adler32-zlib) [ 1234.364645] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 1234.459201] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1236.047144] Key type lgssc registered [ 1236.460112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1239.183492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1239.190547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.528552] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1244.084962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1244.273978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1245.517355] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1248.258709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1248.373757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1250.151710] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1253.020850] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1253.025852] Lustre: MGS: Client 2747f856-faff-4a22-8543-faed9a595fd0 (at 0@lo) reconnecting [ 1253.133936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1253.176780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1254.426056] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1263.584110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1263.586350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.590031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1264.609987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.610946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1264.616176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1264.617878] Lustre: Skipped 1 previous similar message [ 1268.236346] Lustre: server umount lustre-OST0000 complete [ 1269.729255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.729587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1269.733784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1269.738077] Lustre: Skipped 1 previous similar message [ 1274.849176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1274.852519] Lustre: Skipped 1 previous similar message [ 1275.654603] Lustre: server umount lustre-MDT0000 complete [ 1277.041757] Lustre: server umount lustre-MDT0001 complete [ 1278.301682] Lustre: server umount MGS complete [ 1282.012034] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:03:23 (1772759003) [ 1291.532699] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 1294.771810] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1306.700818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1308.002030] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1311.900160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1311.992271] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1312.009520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1312.045758] Lustre: lustre-MDT0001: new disk, initializing [ 1312.073699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1312.083138] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1312.085858] Lustre: 88525:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1312.088762] LustreError: 88525:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1313.372085] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1317.538352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1317.647989] Lustre: lustre-OST0000: new disk, initializing [ 1317.649801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1317.671256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.582260] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1322.399882] Lustre: 89745:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000c75c653b x1858872495080192/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1772759060 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1322.703935] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1324.953658] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1324.958270] Lustre: MGS: Client 9a2745c7-5101-4020-b45b-eedf4d605aea (at 0@lo) reconnecting [ 1324.972069] Lustre: 90207:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1324.980024] LustreError: 88510:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1324.982995] LustreError: 88510:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1324.988293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1325.024753] Lustre: lustre-MDT0000: new disk, initializing [ 1325.079416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1326.112756] Lustre: 90234:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000088390f43 x1858872495085824/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1772759063 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1326.175557] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1326.384208] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1331.169984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1331.172713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1331.239668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1350.624667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.626027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1350.629198] Lustre: Skipped 2 previous similar messages [ 1356.368757] Lustre: server umount lustre-OST0000 complete [ 1360.865976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1360.866357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.877428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1360.879191] Lustre: Skipped 4 previous similar messages [ 1363.976809] Lustre: server umount lustre-MDT0000 complete [ 1371.431570] Lustre: server umount lustre-MDT0001 complete [ 1372.733427] Lustre: server umount MGS complete [ 1376.592579] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:04:57 (1772759097) [ 1382.159927] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1383.153834] Key type lgssc unregistered [ 1383.280363] LNet: 92695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.282268] LNetError: 92695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.289318] LNet: Removed LNI 192.168.203.119@tcp [ 1383.582673] Key type .llcrypt unregistered [ 1383.583558] Key type ._llcrypt unregistered [ 1390.213972] Key type ._llcrypt registered [ 1390.215084] Key type .llcrypt registered [ 1390.255269] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1390.787623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1390.797774] alg: No test for adler32 (adler32-zlib) [ 1391.655720] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 1391.750309] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1393.327137] Key type lgssc registered [ 1393.679147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1396.195079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1396.200868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.359975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.629237] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1401.392647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1401.511579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1402.813147] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1404.624266] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1405.809908] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1408.052377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1408.170096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1410.080747] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1411.888081] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1422.501788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1433.057627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1433.057642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1433.059250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1433.061630] Lustre: Skipped 1 previous similar message [ 1436.113105] Lustre: server umount lustre-OST0000 complete [ 1437.663740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1437.666061] LustreError: Skipped 1 previous similar message [ 1437.667330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1437.671287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1437.672925] Lustre: Skipped 1 previous similar message [ 1443.296519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1443.298640] Lustre: Skipped 3 previous similar messages [ 1443.483084] Lustre: server umount lustre-MDT0000 complete [ 1444.728116] LustreError: 95488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772759166 with bad export cookie 2137896708833644906 [ 1444.730943] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1444.731626] LustreError: 95488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1444.844372] Lustre: server umount lustre-MDT0001 complete [ 1448.601731] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1449.617713] Key type lgssc unregistered [ 1449.744383] LNet: 97069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.747428] LNetError: 97069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1449.764380] LNet: Removed LNI 192.168.203.119@tcp [ 1450.057560] Key type .llcrypt unregistered [ 1450.059063] Key type ._llcrypt unregistered [ 1456.316980] Key type ._llcrypt registered [ 1456.318150] Key type .llcrypt registered [ 1456.352261] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1456.804692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1456.841813] alg: No test for adler32 (adler32-zlib) [ 1457.698821] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 1457.790114] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1459.375167] Key type lgssc registered [ 1459.758663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1461.853926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1461.855046] LustreError: 98111:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 1461.855976] Lustre: Skipped 1 previous similar message [ 1461.860629] LustreError: 98112:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 1461.862516] LustreError: 98111:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1461.866743] LustreError: 98111:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1461.871282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.872376] LustreError: 98111:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1463.048862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.552328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1465.652989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1466.978299] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1468.591972] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing lsmod [ 1469.559906] LustreError: 99281:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 1469.560062] LustreError: 99282:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 1469.564127] LustreError: 99281:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 1469.566521] LustreError: 99281:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1469.569588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1469.570457] LustreError: 99281:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1469.713363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1470.663535] LustreError: 98179:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1470.666709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1470.668271] LustreError: 98179:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1470.668660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1470.674112] Lustre: Skipped 2 previous similar messages [ 1470.677651] LustreError: 98179:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1475.809081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1475.811748] Lustre: Skipped 4 previous similar messages [ 1476.826156] Lustre: server umount lustre-MDT0000 complete [ 1478.108251] LustreError: 98125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772759199 with bad export cookie 6332435296409878662 [ 1478.111560] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1478.112081] LustreError: 98125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1479.903215] LustreError: 98580:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b3d72fdd x1858872728893952/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1479.951420] Lustre: server umount lustre-MDT0001 complete [ 1490.526357] Lustre: server umount lustre-OST0000 complete [ 1493.348274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1493.504645] LustreError: 100350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1493.511203] LustreError: 100350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1493.529976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.888768] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1497.765567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1497.897407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1499.243108] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1501.093666] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1502.398185] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1504.660182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1506.671393] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1508.478693] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1519.142327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1523.169431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1523.172123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.176285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1524.194156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1524.199408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1526.538747] Lustre: server umount lustre-OST0000 complete [ 1528.799906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1528.804343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1528.810433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1528.812333] Lustre: Skipped 2 previous similar messages [ 1534.432313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1534.434949] Lustre: Skipped 3 previous similar messages [ 1542.111114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1542.167043] Lustre: server umount lustre-MDT0000 complete [ 1543.410158] LustreError: 100331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772759265 with bad export cookie 6332435296409879327 [ 1543.413047] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1543.414563] LustreError: 100331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1543.520709] Lustre: server umount lustre-MDT0001 complete [ 1547.248401] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1548.303987] Key type lgssc unregistered [ 1548.432491] LNet: 103551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.436209] LNetError: 103551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1548.443394] LNet: Removed LNI 192.168.203.119@tcp [ 1548.758259] Key type .llcrypt unregistered [ 1548.759363] Key type ._llcrypt unregistered [ 1557.749538] Key type ._llcrypt registered [ 1557.750749] Key type .llcrypt registered [ 1557.794966] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1558.290640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1558.348663] alg: No test for adler32 (adler32-zlib) [ 1559.195985] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 1559.278917] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1560.855181] Key type lgssc registered [ 1561.213720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1563.749486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1563.756759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.919936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.161665] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1569.006529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.137676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1570.451911] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1572.471037] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1573.820756] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1576.157026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1576.277656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1578.223089] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1580.120121] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1590.631660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1592.846910] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1593.646530] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1595.872953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1595.874873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.879300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1595.881108] Lustre: Skipped 1 previous similar message [ 1596.684573] Lustre: server umount lustre-OST0000 complete [ 1599.968089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1599.971432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1599.974802] Lustre: Skipped 1 previous similar message [ 1599.976605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1600.992610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1600.992792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.069444] Lustre: server umount lustre-MDT0000 complete [ 1605.299656] LustreError: 104909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772759327 with bad export cookie 15180876669006410654 [ 1605.302445] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1605.303890] LustreError: 104909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1605.434541] Lustre: server umount lustre-MDT0001 complete [ 1608.858167] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1609.895576] Key type lgssc unregistered [ 1610.019345] LNet: 108228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.021400] LNetError: 108228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.028259] LNet: Removed LNI 192.168.203.119@tcp [ 1610.329731] Key type .llcrypt unregistered [ 1610.330922] Key type ._llcrypt unregistered [ 1617.192829] Key type ._llcrypt registered [ 1617.193889] Key type .llcrypt registered [ 1617.232133] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1617.597669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1617.659586] alg: No test for adler32 (adler32-zlib) [ 1618.523318] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 1618.610352] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1620.191154] Key type lgssc registered [ 1620.570183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1623.110977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1623.116873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.278791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1625.575168] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1628.544789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1628.667462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1629.978237] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1631.731363] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1632.889084] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1635.073696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1635.187521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1636.928470] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1638.676522] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1649.510201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1659.873040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1659.874126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.874993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1659.884248] Lustre: Skipped 1 previous similar message [ 1663.061575] Lustre: server umount lustre-OST0000 complete [ 1664.480139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1664.482456] LustreError: Skipped 1 previous similar message [ 1664.483828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.487589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1664.489218] Lustre: Skipped 1 previous similar message [ 1670.112335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1670.115370] Lustre: Skipped 2 previous similar messages [ 1670.546854] Lustre: server umount lustre-MDT0000 complete [ 1671.851761] LustreError: 109391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772759393 with bad export cookie 6783255252773698049 [ 1671.855171] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1671.857889] LustreError: 109391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1671.985734] Lustre: server umount lustre-MDT0001 complete [ 1675.743561] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1676.768252] Key type lgssc unregistered [ 1676.902356] LNet: 112609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.904721] LNetError: 112609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.917435] LNet: Removed LNI 192.168.203.119@tcp [ 1677.208037] Key type .llcrypt unregistered [ 1677.209109] Key type ._llcrypt unregistered [ 1686.373904] Key type ._llcrypt registered [ 1686.375547] Key type .llcrypt registered [ 1686.419189] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1686.888406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1686.898416] alg: No test for adler32 (adler32-zlib) [ 1687.759031] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 1687.841239] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1689.423150] Key type lgssc registered [ 1689.801642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1692.432839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1692.442525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1693.630472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.053599] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1697.917087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.036264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1699.304838] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1701.134413] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1702.370673] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1704.560986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1704.672598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.522031] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1708.290236] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1720.038881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1726.403328] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1727.616313] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1727.619058] Lustre: Skipped 2 previous similar messages [ 1729.003537] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1729.006651] Lustre: Skipped 2 previous similar messages [ 1732.092517] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1732.105208] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1735.341519] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.19@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1735.345021] Lustre: Skipped 2 previous similar messages [ 1740.299409] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.19@tcp 192.168.0.[2,10]@tcp [ 1742.870890] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.19/24@tcp in log lustre-MDT0000 [ 1742.873795] Lustre: Skipped 2 previous similar messages [ 1746.233382] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.19/24@tcp [ 1746.236148] Lustre: Skipped 1 previous similar message [ 1754.080126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1754.082452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.086095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.129776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1756.130419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.131771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.137515] Lustre: Skipped 1 previous similar message [ 1757.390817] Lustre: server umount lustre-OST0000 complete [ 1759.711893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1759.714124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1759.717872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1764.763773] Lustre: server umount lustre-MDT0000 complete [ 1765.981300] LustreError: 115603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772759487 with bad export cookie 2744460743094309394 [ 1765.984111] LustreError: 115603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1765.984639] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1766.105877] Lustre: server umount lustre-MDT0001 complete [ 1769.690104] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1770.706589] Key type lgssc unregistered [ 1770.822278] LNet: 118689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.825204] LNetError: 118689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1770.831296] LNet: Removed LNI 192.168.203.119@tcp [ 1771.128871] Key type .llcrypt unregistered [ 1771.130397] Key type ._llcrypt unregistered [ 1781.428534] Key type ._llcrypt registered [ 1781.429991] Key type .llcrypt registered [ 1781.467915] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1781.795762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1781.815197] alg: No test for adler32 (adler32-zlib) [ 1782.661642] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 1782.742948] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1784.327144] Key type lgssc registered [ 1784.676132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1787.346089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1787.352972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1788.512336] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1788.517678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1788.522216] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.19/24@tcp [ 1789.783453] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1792.600206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1792.709517] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1792.717995] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.19/24@tcp [ 1792.734522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1794.105142] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1795.917771] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1797.083461] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1799.311905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1799.451410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.370421] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1803.164403] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1813.805645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 1823.200263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1823.202672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.207215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1824.226106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1824.227497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1824.227950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1824.234478] Lustre: Skipped 1 previous similar message [ 1827.732995] Lustre: server umount lustre-OST0000 complete [ 1829.345034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.345543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1829.348167] Lustre: Skipped 1 previous similar message [ 1834.464192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1834.465955] Lustre: Skipped 2 previous similar messages [ 1835.098519] Lustre: server umount lustre-MDT0000 complete [ 1836.399583] LustreError: 120192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772759558 with bad export cookie 15893292515912157109 [ 1836.403084] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1836.403502] LustreError: 120192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1836.538951] Lustre: server umount lustre-MDT0001 complete [ 1840.251690] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1841.367298] Key type lgssc unregistered [ 1841.499392] LNet: 123464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.501580] LNetError: 123464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.510357] LNet: Removed LNI 192.168.203.119@tcp [ 1841.833559] Key type .llcrypt unregistered [ 1841.835520] Key type ._llcrypt unregistered [ 1850.420083] Key type ._llcrypt registered [ 1850.421162] Key type .llcrypt registered [ 1850.459866] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 1856.496945] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1856.880946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1856.934460] alg: No test for adler32 (adler32-zlib) [ 1857.802670] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 1857.891932] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1859.471086] Key type lgssc registered [ 1859.837971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1871.249345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1871.254724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1872.343527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1872.354411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1872.395577] Lustre: lustre-MDT0000: new disk, initializing [ 1872.418472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.424930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1873.710855] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1877.883910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.929090] Lustre: 126801:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1877.959726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1877.962082] Lustre: Skipped 1 previous similar message [ 1878.004299] Lustre: lustre-MDT0001: new disk, initializing [ 1878.045485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1878.059875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1878.063287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1879.418705] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1882.482362] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1883.678343] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1885.902907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1886.008376] Lustre: lustre-OST0000: new disk, initializing [ 1886.010394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1886.038309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.097052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1887.102296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1887.136699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1887.899652] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1890.988231] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1892.287197] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1892.356610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1893.647074] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1893.717370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1897.441964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.442874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1897.447483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1897.451147] Lustre: Skipped 1 previous similar message [ 1900.431717] Lustre: server umount lustre-OST0000 complete [ 1902.560778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.561469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1902.564648] Lustre: Skipped 2 previous similar messages [ 1907.680172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1907.681833] Lustre: Skipped 2 previous similar messages [ 1907.802794] Lustre: server umount lustre-MDT0000 complete [ 1909.003173] LustreError: 126793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772759630 with bad export cookie 6646454028953410472 [ 1909.006124] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1909.006484] LustreError: 126793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1909.120750] Lustre: server umount lustre-MDT0001 complete [ 1911.903702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.072737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1913.337048] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1916.190421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1916.312747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1917.652586] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1919.430549] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1920.597114] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1922.832233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1924.751081] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1926.568027] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1927.900324] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1937.150348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 1938.504107] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1938.575786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1941.938808] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1942.005471] Lustre: lustre-OST0001: new disk, initializing [ 1942.007239] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1942.038324] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1942.040450] Lustre: Skipped 1 previous similar message [ 1943.919956] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1947.190622] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1952.258898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1952.262321] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1952.289583] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1952.359136] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 1953.658262] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1953.727070] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1954.902481] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88fe45f3a800.ost_server_uuid 50 [ 1955.371510] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88fe45f3a800.ost_server_uuid in FULL state after 0 sec [ 1957.345794] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1957.347971] LustreError: Skipped 1 previous similar message [ 1957.349066] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.350301] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1957.352372] Lustre: Skipped 1 previous similar message [ 1957.355922] Lustre: Skipped 1 previous similar message [ 1962.468432] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1962.471843] Lustre: Skipped 1 previous similar message [ 1963.260123] Lustre: server umount lustre-OST0001 complete [ 1967.072075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1967.074159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.586523] LustreError: 134432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1967.593417] LustreError: 134432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1970.574584] Lustre: server umount lustre-OST0000 complete [ 1972.706199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1972.706819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.709114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1972.713792] Lustre: Skipped 2 previous similar messages [ 1972.715416] Lustre: Skipped 2 previous similar messages [ 1977.950909] Lustre: server umount lustre-MDT0000 complete [ 1979.183176] LustreError: 130593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772759700 with bad export cookie 6646454028953411340 [ 1979.186523] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1979.187271] LustreError: 130593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1979.305730] Lustre: server umount lustre-MDT0001 complete [ 1985.508520] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1986.475377] Key type lgssc unregistered [ 1986.593359] LNet: 136545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1986.595710] LNetError: 136545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1986.603322] LNet: Removed LNI 192.168.203.119@tcp [ 1986.887408] Key type .llcrypt unregistered [ 1986.888578] Key type ._llcrypt unregistered [ 1999.668468] Key type ._llcrypt registered [ 1999.669588] Key type .llcrypt registered [ 1999.706812] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 2005.615363] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2006.047779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2006.058772] alg: No test for adler32 (adler32-zlib) [ 2006.905594] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 2006.992272] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2008.575162] Key type lgssc registered [ 2008.939386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2020.105286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2020.111151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.201179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2021.215981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2021.266624] Lustre: lustre-MDT0000: new disk, initializing [ 2021.292995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.302313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2022.626642] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2026.744638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.785604] Lustre: 140634:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2026.807094] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2026.808846] Lustre: Skipped 1 previous similar message [ 2026.842783] Lustre: lustre-MDT0001: new disk, initializing [ 2026.873795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2026.886400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2026.889536] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2028.145938] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2031.235089] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2032.397289] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2034.608820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2034.709677] Lustre: lustre-OST0000: new disk, initializing [ 2034.711632] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2034.740265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.764802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2035.768347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2035.799281] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2036.583965] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2039.612374] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2041.627966] Lustre: Failing over lustre-OST0000 [ 2041.693375] Lustre: server umount lustre-OST0000 complete [ 2054.525193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2054.642150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2054.646438] Lustre: Skipped 1 previous similar message [ 2054.650647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.659591] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2056.252377] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2056.534417] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2057.807921] Lustre: Failing over lustre-MDT0000 [ 2057.908667] Lustre: server umount lustre-MDT0000 complete [ 2059.745421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.749914] Lustre: Skipped 2 previous similar messages [ 2061.791141] Lustre: 138862:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772759767/real 1772759767] req@000000003aeb7c08 x1858873304575232/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772759783 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2061.800803] Lustre: 138862:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2064.864948] LustreError: 141429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2064.870393] LustreError: 141429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2066.483426] LustreError: 140642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.911163] Lustre: 138864:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772759772/real 1772759772] req@000000004a64a573 x1858873304576384/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772759788 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2069.983994] LustreError: 140645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2069.988879] LustreError: 140645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2070.735701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.780661] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2070.785126] Lustre: 140637:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.203.119@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2070.861103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2070.881301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.882917] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2070.883602] Lustre: lustre-OST0000: Recovery over after 0:14, of 3 clients 3 recovered and 0 were evicted. [ 2070.926438] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2071.604213] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2072.174046] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2076.133570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2076.137237] Lustre: Skipped 1 previous similar message [ 2076.144203] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2076.162785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2081.253354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.253940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2081.257986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2081.259556] LustreError: Skipped 1 previous similar message [ 2083.287835] Lustre: server umount lustre-OST0000 complete [ 2086.369602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2086.369888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.371209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2086.371217] Lustre: Skipped 1 previous similar message [ 2086.378238] Lustre: Skipped 2 previous similar messages [ 2090.781263] Lustre: server umount lustre-MDT0000 complete [ 2091.488438] LustreError: 141429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.492746] LustreError: 141429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2092.014144] LustreError: 140625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772759813 with bad export cookie 12495254453533860711 [ 2092.017632] LustreError: 140625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2092.018079] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2092.138170] Lustre: server umount lustre-MDT0001 complete [ 2095.534884] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2096.521663] Key type lgssc unregistered [ 2096.646355] LNet: 145860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.648671] LNetError: 145860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2096.656322] LNet: Removed LNI 192.168.203.119@tcp [ 2096.943670] Key type .llcrypt unregistered [ 2096.945358] Key type ._llcrypt unregistered [ 2116.601305] Key type ._llcrypt registered [ 2116.602763] Key type .llcrypt registered [ 2116.648680] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2117.221714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2117.233404] alg: No test for adler32 (adler32-zlib) [ 2118.089443] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 2118.172461] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2119.751110] Key type lgssc registered [ 2120.117316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2122.628298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2122.635068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.799676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2125.118032] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2128.193426] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2130.199224] LustreError: 147388:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2130.205090] LustreError: 147388:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2130.208344] LustreError: 147388:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2130.322513] Lustre: server umount lustre-MDT0000 complete [ 2135.078351] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 2138.321255] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2149.893894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.985938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2149.998331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2150.036063] Lustre: lustre-MDT0000: new disk, initializing [ 2150.059977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.068306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2151.373374] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2155.452888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2155.500098] Lustre: 150536:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2155.525416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2155.528822] Lustre: Skipped 1 previous similar message [ 2155.577315] Lustre: lustre-MDT0001: new disk, initializing [ 2155.611656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2155.623961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2155.629189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2156.938255] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2159.979293] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2161.124086] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2163.382448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2163.488848] Lustre: lustre-OST0000: new disk, initializing [ 2163.490801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2163.518543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.055105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2165.059119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2165.100070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2165.411994] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2168.541751] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2170.407457] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2171.051220] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:18:12 (1772759892) [ 2236.897172] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.897423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.901207] Lustre: Skipped 2 previous similar messages [ 2236.905731] Lustre: Skipped 2 previous similar messages [ 2240.428911] Lustre: server umount lustre-MDT0000 complete [ 2241.696088] LustreError: 151313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772759963 with bad export cookie 4324793680683504892 [ 2241.698553] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2241.701264] LustreError: 151313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2242.015647] LustreError: 151666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.015689] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.023072] LustreError: 151666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2242.028182] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2246.047733] LustreError: 152885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2246.047759] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2251.231426] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2251.231427] LustreError: 150543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.839151] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2255.902820] Lustre: server umount lustre-MDT0001 complete [ 2263.251843] Lustre: server umount lustre-OST0000 complete [ 2266.124076] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 2269.342913] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2281.482114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2281.576173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2281.589426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2281.629383] Lustre: lustre-MDT0000: new disk, initializing [ 2281.653849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2281.660441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2282.937879] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2287.014741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.055077] Lustre: 156762:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2287.079636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2287.082161] Lustre: Skipped 1 previous similar message [ 2287.122481] Lustre: lustre-MDT0001: new disk, initializing [ 2287.166198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2287.169535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2288.459164] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2291.474274] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2292.644455] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2294.877558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2295.004170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.006124] Lustre: Skipped 1 previous similar message [ 2296.061740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2296.065934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2296.104505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2296.867245] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2299.978595] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2301.298618] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2301.397004] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2303.041234] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2303.118303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2305.868361] Lustre: server umount lustre-OST0000 complete [ 2307.552030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2307.554490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.558322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2307.560023] Lustre: Skipped 1 previous similar message [ 2311.650407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.768788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.770921] Lustre: Skipped 3 previous similar messages [ 2321.375115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2321.431237] Lustre: server umount lustre-MDT0000 complete [ 2321.888612] LustreError: 157811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.892638] LustreError: 157811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2322.911146] Lustre: 146671:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772760028/real 1772760028] req@00000000d757a729 x1858873421596672/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1772760044 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2322.916751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.135857] LustreError: 156753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772760044 with bad export cookie 4324793680683663708 [ 2323.139360] LustreError: 156753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2323.140838] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.277985] Lustre: server umount lustre-MDT0001 complete [ 2325.890242] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:20:47 (1772760047) [ 2331.801446] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 2335.747699] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2353.736169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.884260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2353.902828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2353.905218] Lustre: Skipped 1 previous similar message [ 2353.956857] Lustre: lustre-MDT0000: new disk, initializing [ 2353.959238] Lustre: Skipped 1 previous similar message [ 2354.015853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2354.028565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2356.121183] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2361.981495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2362.034078] Lustre: 163074:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2362.144660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2364.057402] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2368.212621] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2370.154503] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2373.473153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2373.619453] Lustre: lustre-OST0000: new disk, initializing [ 2373.621629] Lustre: Skipped 1 previous similar message [ 2373.624330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2373.629790] Lustre: Skipped 2 previous similar messages [ 2375.821483] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2375.971893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2378.312220] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2385.884468] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2387.892747] hrtimer: interrupt took 8442764 ns [ 2396.644271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2396.644639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.648510] LustreError: Skipped 1 previous similar message [ 2396.656670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2396.662892] Lustre: Skipped 1 previous similar message [ 2399.793670] Lustre: server umount lustre-OST0000 complete [ 2416.095147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2416.180897] Lustre: server umount lustre-MDT0000 complete [ 2417.122308] LustreError: 163078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.128138] LustreError: 163078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2417.992303] LustreError: 163066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772760139 with bad export cookie 4324793680683664576 [ 2417.995523] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2417.997266] LustreError: 163066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2418.146846] Lustre: server umount lustre-MDT0001 complete [ 2422.002274] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:22:22 (1772760142) [ 2429.188849] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 2434.280678] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2453.948125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2454.237513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2454.261638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2454.333361] Lustre: lustre-MDT0000: new disk, initializing [ 2454.402063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2454.407568] Lustre: Skipped 2 previous similar messages [ 2454.419881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2454.425437] Lustre: Skipped 2 previous similar messages [ 2457.928938] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2468.961398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.064311] Lustre: 168992:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2469.237685] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2473.342756] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2481.507645] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2485.607169] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2492.008404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2500.528755] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2500.815862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2506.805929] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2513.804302] Lustre: server umount lustre-OST0000 complete [ 2516.455430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.457524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2516.465379] Lustre: Skipped 4 previous similar messages [ 2516.486234] Lustre: Skipped 7 previous similar messages [ 2525.741677] Lustre: server umount lustre-MDT0000 complete [ 2526.688877] LustreError: 169004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.696235] LustreError: 169004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2528.051780] LustreError: 168983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772760249 with bad export cookie 4324793680683665661 [ 2528.055089] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2528.059363] LustreError: 168983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2537.552787] Lustre: server umount lustre-MDT0001 complete [ 2544.011231] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2545.744601] Key type lgssc unregistered [ 2545.961839] LNet: 172465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2545.966634] LNetError: 172465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2546.985473] LNet: Removed LNI 192.168.203.119@tcp [ 2547.533502] Key type .llcrypt unregistered [ 2547.535332] Key type ._llcrypt unregistered [ 2565.638649] Key type ._llcrypt registered [ 2565.639810] Key type .llcrypt registered [ 2565.703876] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2566.475378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2566.554180] alg: No test for adler32 (adler32-zlib) [ 2567.582670] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 2567.742462] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2569.415661] Key type lgssc registered [ 2570.159900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2576.294831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2576.310268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2577.724031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.179562] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2587.855410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.122287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2591.277220] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2595.659449] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2598.335363] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2603.178330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2603.459365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.385562] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2608.636560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2612.692620] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2633.190763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.192910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2633.197859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2633.216576] Lustre: Skipped 1 previous similar message [ 2636.091350] Lustre: server umount lustre-OST0000 complete [ 2639.333864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2639.340280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.348621] Lustre: Skipped 1 previous similar message [ 2639.353499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2643.426418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2643.428170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2648.551494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2648.568520] Lustre: Skipped 2 previous similar messages [ 2653.151149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2653.420599] Lustre: server umount lustre-MDT0000 complete [ 2653.665509] LustreError: 173845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.684598] LustreError: 173845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2656.421245] LustreError: 173825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772760378 with bad export cookie 3955018280088355113 [ 2656.444121] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2656.700842] Lustre: server umount lustre-MDT0001 complete [ 2665.603386] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2667.511750] Key type lgssc unregistered [ 2667.758813] LNet: 177043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2667.765969] LNetError: 177043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2668.837764] LNet: Removed LNI 192.168.203.119@tcp [ 2669.514108] Key type .llcrypt unregistered [ 2669.516058] Key type ._llcrypt unregistered [ 2688.399811] Key type ._llcrypt registered [ 2688.402953] Key type .llcrypt registered [ 2688.483599] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2689.201638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2689.294875] alg: No test for adler32 (adler32-zlib) [ 2690.335420] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 2690.550153] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2692.199795] Key type lgssc registered [ 2693.442839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2699.941550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2699.958227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2701.311300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.835873] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2711.526753] LustreError: 178424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.554651] LustreError: 178424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2713.328167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.804937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2717.303969] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2722.119471] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2725.360820] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2730.441105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2730.723167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2735.458582] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2739.758478] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2747.138612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2756.688515] Lustre: lustre-OST0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 2757.606663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.609775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2757.616423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2757.631075] Lustre: Skipped 1 previous similar message [ 2761.271261] Lustre: server umount lustre-OST0000 complete [ 2765.784188] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff88fe477f5000.ost_server_uuid 50 [ 2767.157281] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2780.639977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2780.646529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2780.659869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2780.670263] Lustre: Skipped 1 previous similar message [ 2783.201267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2783.202626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2785.299446] Lustre: server umount lustre-MDT0000 complete [ 2788.255835] LustreError: 178404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772760509 with bad export cookie 13219115965272208769 [ 2788.260319] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2788.274763] LustreError: 178404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2788.557500] Lustre: server umount lustre-MDT0001 complete [ 2796.111371] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:28:36 (1772760516) [ 2802.030685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.490367] LustreError: 181928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2802.580441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.455403] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2807.781363] LustreError: 181929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.901939] LustreError: 181928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2814.023866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.259542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2818.280871] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2823.468177] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2826.683902] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2832.925566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2833.373255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.361173] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2843.466053] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2847.728217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2849.261285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2849.428288] Lustre: lustre-OST0001: new disk, initializing [ 2849.436280] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2849.513422] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2855.286743] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2857.023832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2857.031621] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2857.125920] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2862.467931] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2867.687879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2867.692909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2867.707204] Lustre: Skipped 1 previous similar message [ 2867.720887] Lustre: Skipped 2 previous similar messages [ 2871.259748] Lustre: server umount lustre-OST0000 complete [ 2872.804575] LustreError: 183521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2872.820992] LustreError: 183521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2875.861174] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2876.038237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2879.708314] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2879.891768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2883.043981] LustreError: 184882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.076020] LustreError: 184882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2893.282097] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2893.284376] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.286169] LustreError: Skipped 1 previous similar message [ 2893.290334] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2893.301038] Lustre: Skipped 1 previous similar message [ 2893.305032] Lustre: Skipped 1 previous similar message [ 2898.062795] Lustre: server umount lustre-OST0001 complete [ 2901.473837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2901.487564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2906.822770] Lustre: server umount lustre-MDT0000 complete [ 2908.642251] LustreError: 184474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.654193] LustreError: 184474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2909.317020] LustreError: 181910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772760631 with bad export cookie 13219115965272209833 [ 2909.318165] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2909.328069] LustreError: 181910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2909.518419] Lustre: server umount lustre-MDT0001 complete [ 2925.334956] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:30:46 (1772760646) [ 2929.716794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.965383] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2929.991290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2930.129417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2932.781908] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2938.496423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2938.561594] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2938.574890] Lustre: 187722:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2941.261269] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2944.517850] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2946.770033] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2950.733386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2950.866568] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2954.063544] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2957.043208] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2957.869960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2961.144845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2961.196241] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2964.504888] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2967.830703] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2973.152737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2973.159450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.165615] Lustre: Skipped 1 previous similar message [ 2973.168561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2973.172607] Lustre: Skipped 4 previous similar messages [ 2975.588174] Lustre: server umount lustre-OST0000 complete [ 2978.357571] LustreError: 189343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2989.538846] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2989.542498] LustreError: Skipped 2 previous similar messages [ 2993.643462] Lustre: server umount lustre-OST0001 complete [ 3001.735276] Lustre: server umount lustre-MDT0000 complete [ 3003.765223] LustreError: 190266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772760725 with bad export cookie 13219115965272211247 [ 3003.769906] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3003.771761] LustreError: 190266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3003.922882] Lustre: server umount lustre-MDT0001 complete [ 3017.351690] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:32:18 (1772760738) [ 3025.922113] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 3030.368933] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3047.333077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3047.471574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3047.490175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3047.530621] Lustre: lustre-MDT0000: new disk, initializing [ 3047.571914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.576269] Lustre: Skipped 3 previous similar messages [ 3047.587236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3049.613564] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3055.294417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.349776] Lustre: 195299:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3055.384390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3055.387422] Lustre: Skipped 1 previous similar message [ 3055.427898] Lustre: lustre-MDT0001: new disk, initializing [ 3055.491442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3055.496736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3057.154682] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3060.930841] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3062.553775] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3065.769993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3065.898694] Lustre: lustre-OST0000: new disk, initializing [ 3065.901464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3068.490705] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3072.012991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3072.022927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3072.069537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3072.362787] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3074.201257] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3074.285955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3076.186891] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3076.304337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3079.295520] Lustre: server umount lustre-OST0000 complete [ 3082.208783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.213182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3082.214533] Lustre: Skipped 6 previous similar messages [ 3082.220562] Lustre: Skipped 7 previous similar messages [ 3090.439926] Lustre: server umount lustre-MDT0000 complete [ 3092.354780] LustreError: 195291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772760814 with bad export cookie 13219115965272212808 [ 3092.355454] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3092.361326] LustreError: 195291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3102.764950] Lustre: server umount lustre-MDT0001 complete [ 3107.089929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.298462] LustreError: 199109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.308362] LustreError: 199109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3109.212639] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3113.371734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.573038] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3118.203727] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3120.110687] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3123.715372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3126.599976] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3129.405635] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3131.776217] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3133.992527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3136.414801] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3136.555823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3138.532740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3139.828078] Lustre: server umount lustre-OST0000 complete [ 3143.502294] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3143.608732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3146.206813] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3171.420253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3174.449049] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3177.455384] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3178.794140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3178.804086] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3178.811968] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3193.093494] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 14 sec [ 3199.460561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3199.460696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.471244] Lustre: Skipped 2 previous similar messages [ 3199.474639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3199.476981] Lustre: Skipped 4 previous similar messages [ 3201.042547] Lustre: server umount lustre-OST0000 complete [ 3217.375874] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3217.474091] Lustre: server umount lustre-MDT0000 complete [ 3219.866142] LustreError: 199090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772760941 with bad export cookie 13219115965272213662 [ 3219.868540] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3219.875893] LustreError: 199090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3220.062742] Lustre: server umount lustre-MDT0001 complete [ 3224.815708] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:35:45 (1772760945) [ 3229.442028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3229.735849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3229.743569] Lustre: Skipped 6 previous similar messages [ 3232.268838] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3237.189636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.441804] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3242.308298] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3244.072985] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3247.517476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3247.587422] LustreError: 206206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3247.599605] LustreError: 206206:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3250.729896] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3253.754742] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3255.929153] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3256.051509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3258.235738] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3258.350125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3261.555632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3261.639926] Lustre: lustre-OST0001: new disk, initializing [ 3261.642813] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3264.207447] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3268.045533] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3269.827970] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3270.160165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3270.166264] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3270.200768] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3270.940826] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3272.672886] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3272.775613] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3275.232081] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3275.235632] LustreError: Skipped 1 previous similar message [ 3279.770272] Lustre: server umount lustre-OST0001 complete [ 3282.830775] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3282.948748] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3284.816775] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3284.911584] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3309.788269] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3312.636380] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3315.176402] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3315.182981] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3315.183136] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3315.193583] Lustre: Skipped 2 previous similar messages [ 3315.414589] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3316.264630] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3323.276375] Lustre: server umount lustre-OST0001 complete [ 3336.673419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3336.675560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3336.681029] Lustre: Skipped 8 previous similar messages [ 3336.689072] Lustre: Skipped 12 previous similar messages [ 3342.499914] Lustre: server umount lustre-OST0000 complete [ 3350.590989] Lustre: server umount lustre-MDT0000 complete [ 3352.454019] LustreError: 207612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772761074 with bad export cookie 13219115965272214614 [ 3352.455199] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3352.458169] LustreError: 207612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3358.661729] Lustre: server umount lustre-MDT0001 complete [ 3370.316139] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:38:11 (1772761091) [ 3373.666632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3373.803889] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3373.820221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3375.965667] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3380.391515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3380.448691] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3380.462047] Lustre: 212565:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3382.464067] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3385.582424] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3387.611855] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3391.604977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3391.742505] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3395.025412] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3398.009706] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3403.127914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3403.197577] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3406.011432] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3408.617639] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3409.508423] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3411.422859] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3411.528284] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3413.365647] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3413.469122] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3415.325423] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88fe44107800.ost_server_uuid 50 [ 3416.110109] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88fe44107800.ost_server_uuid in FULL state after 0 sec [ 3416.600760] Lustre: Permanently deactivating lustre-OST0001 [ 3416.610364] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3417.232673] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3417.859197] Lustre: Permanently reactivating lustre-OST0001 [ 3425.348083] Lustre: server umount lustre-OST0001 complete [ 3428.321386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3428.328359] LustreError: Skipped 6 previous similar messages [ 3433.509405] Lustre: server umount lustre-OST0000 complete [ 3446.279636] Lustre: server umount lustre-MDT0000 complete [ 3447.263600] LustreError: 216692:0:(osp_dev.c:694:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 3448.239309] LustreError: 213289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772761169 with bad export cookie 13219115965272215902 [ 3448.248322] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3448.254813] LustreError: 212567:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3448.410852] Lustre: server umount lustre-MDT0001 complete [ 3460.283208] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:39:41 (1772761181) [ 3464.217985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.447165] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3464.467154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3464.472270] Lustre: Skipped 5 previous similar messages [ 3466.360094] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3470.555801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3470.621465] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3470.639835] Lustre: 218672:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3472.734336] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3475.257345] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3477.016915] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3480.817666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3480.975238] Lustre: Permanently deactivating lustre-OST0000 [ 3480.982960] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3481.548618] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3481.552392] Lustre: Skipped 1 previous similar message [ 3484.159406] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3487.332672] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3491.181261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3491.339316] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3491.343802] Lustre: Skipped 11 previous similar messages [ 3494.506195] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3497.467434] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3497.594821] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3498.455229] Lustre: Permanently reactivating lustre-OST0000 [ 3498.464306] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3498.467153] Lustre: Skipped 2 previous similar messages [ 3505.686926] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3505.692484] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3505.700842] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3505.726034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3505.737930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3516.808495] Lustre: server umount lustre-OST0001 complete [ 3518.436394] LustreError: 220379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.443454] LustreError: 220379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 32 previous similar messages [ 3524.566868] Lustre: server umount lustre-OST0000 complete [ 3532.340563] Lustre: server umount lustre-MDT0000 complete [ 3534.107996] LustreError: 220313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772761255 with bad export cookie 13219115965272217610 [ 3534.110450] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3534.117342] LustreError: 220313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3534.275107] Lustre: server umount lustre-MDT0001 complete [ 3539.202895] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 3540.691820] Key type lgssc unregistered [ 3540.872513] LNet: 223075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3540.880996] LNetError: 223075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3540.893529] LNet: Removed LNI 192.168.203.119@tcp [ 3541.348841] Key type .llcrypt unregistered [ 3541.350911] Key type ._llcrypt unregistered [ 3552.902198] Key type ._llcrypt registered [ 3552.903878] Key type .llcrypt registered [ 3552.960330] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3553.390310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3553.428346] alg: No test for adler32 (adler32-zlib) [ 3554.355530] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 3554.499375] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 3556.103228] Key type lgssc registered [ 3556.666461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3561.478684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3561.488933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.724689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3564.444491] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3568.507547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3568.559977] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3568.576154] Lustre: 224491:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3568.581471] Lustre: Permanently deactivating lustre-MDT0001 [ 3568.588577] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3568.721516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3570.466612] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3572.951529] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3573.746355] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3574.715970] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3577.725317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3577.904211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3578.914724] LustreError: 226252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3578.920135] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3578.922536] LustreError: 226252:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3580.533690] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3583.169670] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3583.973769] LustreError: 226116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3586.916498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3587.058190] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3588.079142] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3588.079182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3588.139864] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3589.767239] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3592.771932] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3595.773199] Lustre: Permanently reactivating lustre-MDT0001 [ 3595.782118] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3595.784477] Lustre: Skipped 2 previous similar messages [ 3596.578686] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.595388] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3596.603946] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3596.614746] LustreError: 227620:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3596.618370] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3596.622265] LustreError: 227620:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3596.622403] LustreError: 227620:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.203.119@tcp: cfg command failed: rc = -108 [ 3596.622423] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3596.622423] [ 3596.625223] LustreError: 224495:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3600.490058] Lustre: lustre-MDT0001: Client 6413e81a-594c-454e-817c-81e147e60a8e (at 192.168.203.19@tcp) reconnecting [ 3603.566104] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3605.024280] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3608.763552] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3608.919271] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3611.666196] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3611.769955] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3613.162926] Lustre: Permanently deactivating lustre-MDT0001 [ 3613.175251] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3613.178776] Lustre: Skipped 2 previous similar messages [ 3622.943352] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3627.936839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3627.939510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3627.943933] Lustre: Skipped 2 previous similar messages [ 3627.949670] Lustre: Skipped 3 previous similar messages [ 3633.123116] LustreError: 226250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3633.132413] LustreError: 226250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3633.147481] Lustre: server umount lustre-MDT0000 complete [ 3635.389180] LustreError: 225238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772761357 with bad export cookie 1476258060290701504 [ 3635.397901] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3635.588917] Lustre: server umount lustre-MDT0001 complete [ 3644.101744] Lustre: server umount lustre-OST0000 complete [ 3649.817855] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:42:50 (1772761370) [ 3655.427263] Lustre: server umount lustre-OST0001 complete [ 3658.673136] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 3663.325320] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3681.356275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3681.482117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3681.486479] Lustre: Skipped 2 previous similar messages [ 3681.499714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3681.543487] Lustre: lustre-MDT0000: new disk, initializing [ 3681.581776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3681.593807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3683.509497] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3689.229815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3689.287439] Lustre: 232193:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3689.325117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3689.328269] Lustre: Skipped 1 previous similar message [ 3689.375865] Lustre: lustre-MDT0001: new disk, initializing [ 3689.451829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3689.457117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3691.444690] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3695.602034] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3697.495312] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3701.068133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3701.197890] Lustre: lustre-OST0000: new disk, initializing [ 3701.199818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3701.235941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3701.240053] Lustre: Skipped 1 previous similar message [ 3703.991829] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3708.270428] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3710.484021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3710.488524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3710.560274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3714.385198] LustreError: 232200:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3719.579669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3719.659744] Lustre: lustre-OST0001: new disk, initializing [ 3719.662590] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3719.701778] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3722.598057] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3724.479125] LustreError: 232200:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3724.500688] LustreError: 234400:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3726.804968] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3727.610693] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3727.891804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3727.895499] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3727.944111] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3734.599132] LustreError: 234400:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3735.623631] Lustre: lustre-OST0001: Not available for connect from 192.168.203.19@tcp (stopping) [ 3738.098335] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3738.099240] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3738.102188] LustreError: Skipped 1 previous similar message [ 3738.111243] Lustre: Skipped 1 previous similar message [ 3738.112584] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3740.728032] Lustre: lustre-OST0001: Not available for connect from 192.168.203.19@tcp (stopping) [ 3740.730840] Lustre: Skipped 1 previous similar message [ 3741.675496] Lustre: server umount lustre-OST0001 complete [ 3743.207064] LustreError: 233916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3743.213145] LustreError: 233916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3748.329754] LustreError: 234536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3748.336016] LustreError: 234536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3756.512777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3756.513301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3756.516911] LustreError: Skipped 1 previous similar message [ 3756.518402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3756.521612] Lustre: Skipped 2 previous similar messages [ 3756.527974] Lustre: Skipped 1 previous similar message [ 3758.566067] LustreError: 234536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3758.574932] LustreError: 234536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3768.801448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3768.805586] Lustre: Skipped 5 previous similar messages [ 3769.312529] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3769.390396] Lustre: server umount lustre-OST0000 complete [ 3771.360081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3771.364085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3777.440645] Lustre: server umount lustre-MDT0000 complete [ 3779.041157] LustreError: 232204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3779.054585] LustreError: 232204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3779.415110] LustreError: 233930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772761501 with bad export cookie 1476258060290704325 [ 3779.417962] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3779.419519] LustreError: 233930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3785.627973] Lustre: server umount lustre-MDT0001 complete [ 3790.502056] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 3791.907863] Key type lgssc unregistered [ 3792.097687] LNet: 236835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3792.101334] LNetError: 236835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3792.110608] LNet: Removed LNI 192.168.203.119@tcp [ 3792.586497] Key type .llcrypt unregistered [ 3792.588897] Key type ._llcrypt unregistered [ 3812.521844] Key type ._llcrypt registered [ 3812.526057] Key type .llcrypt registered [ 3812.587324] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3813.089921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3813.100760] alg: No test for adler32 (adler32-zlib) [ 3814.012222] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 3814.118286] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 3815.711567] Key type lgssc registered [ 3816.201603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3819.695805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3819.711769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.836733] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3820.846825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3820.920730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3822.659289] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3826.708067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3826.762738] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3826.774920] Lustre: 239001:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3826.904584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3828.504826] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3830.832475] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3832.307128] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3835.107649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3835.218256] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3835.313114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3838.043095] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3838.872052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3840.329122] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3885.025784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3885.025931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.027270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3885.029838] LustreError: Skipped 1 previous similar message [ 3885.038260] Lustre: Skipped 1 previous similar message [ 3886.303357] Lustre: server umount lustre-OST0000 complete [ 3888.008846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3892.720465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3892.844920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3894.851331] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3896.793342] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3905.124722] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3905.129774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3905.129969] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3905.136861] Lustre: Skipped 1 previous similar message [ 3931.106546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3931.106706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.109817] LustreError: Skipped 1 previous similar message [ 3931.115326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3931.119179] Lustre: Skipped 2 previous similar messages [ 3936.225936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3936.228342] Lustre: Skipped 1 previous similar message [ 3936.485968] Lustre: server umount lustre-OST0000 complete [ 3939.807966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3939.811425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3939.816066] Lustre: Skipped 1 previous similar message [ 3939.818044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3946.464752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3946.467457] Lustre: Skipped 3 previous similar messages [ 3952.095113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3952.176669] Lustre: server umount lustre-MDT0000 complete [ 3953.665330] LustreError: 238994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772761675 with bad export cookie 11576457194560649228 [ 3953.668477] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3953.671226] LustreError: 238994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3953.813936] Lustre: server umount lustre-MDT0001 complete [ 3958.140830] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 3959.341913] Key type lgssc unregistered [ 3959.483573] LNet: 243951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3959.487328] LNetError: 243951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3959.498470] LNet: Removed LNI 192.168.203.119@tcp [ 3959.826331] Key type .llcrypt unregistered [ 3959.827522] Key type ._llcrypt unregistered [ 3970.280973] Key type ._llcrypt registered [ 3970.282633] Key type .llcrypt registered [ 3970.327070] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3970.727492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3970.777377] alg: No test for adler32 (adler32-zlib) [ 3971.662170] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 3971.757794] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 3973.351155] Key type lgssc registered [ 3973.790705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3976.938664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3976.946901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3978.155072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3979.865749] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3983.541390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.747947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3985.414987] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3987.791731] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3989.362499] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3992.258312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3992.412595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.098100] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3997.552534] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4002.662878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4013.027161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.028508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4013.028732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4013.031477] Lustre: Skipped 1 previous similar message [ 4014.641367] Lustre: server umount lustre-OST0000 complete [ 4018.144814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.154632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4018.158060] Lustre: Skipped 2 previous similar messages [ 4018.531657] Lustre: server umount lustre-MDT0000 complete [ 4020.176265] LustreError: 248253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772761741 with bad export cookie 15287369624447995571 [ 4020.178946] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4020.182064] LustreError: 248253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4020.324905] Lustre: server umount lustre-MDT0001 complete [ 4024.292841] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 4025.453432] Key type lgssc unregistered [ 4025.600459] LNet: 249273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4025.603554] LNetError: 249273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4025.613542] LNet: Removed LNI 192.168.203.119@tcp [ 4025.939593] Key type .llcrypt unregistered [ 4025.940982] Key type ._llcrypt unregistered [ 4033.888459] Key type ._llcrypt registered [ 4033.890113] Key type .llcrypt registered [ 4033.946924] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4034.512664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4034.520827] alg: No test for adler32 (adler32-zlib) [ 4035.415243] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 4035.521262] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 4037.127158] Key type lgssc registered [ 4037.598220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4041.011777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4041.028840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4042.242288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.886141] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4047.562490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4047.731828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4049.347087] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4051.763621] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4053.275581] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4056.047725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4056.184061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4058.506465] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4060.763569] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4066.410239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4077.026583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4077.032710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4077.032789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4077.038257] Lustre: Skipped 1 previous similar message [ 4082.146176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4082.148522] Lustre: Skipped 1 previous similar message [ 4083.037660] Lustre: server umount lustre-OST0000 complete [ 4087.264622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4087.267802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4087.270090] Lustre: Skipped 2 previous similar messages [ 4087.273846] Lustre: Skipped 1 previous similar message [ 4090.723885] Lustre: server umount lustre-MDT0000 complete [ 4092.326711] LustreError: 250437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772761814 with bad export cookie 4670606285287125360 [ 4092.328197] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4092.335366] LustreError: 250437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4092.450287] Lustre: server umount lustre-MDT0001 complete [ 4096.768363] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 4098.006824] Key type lgssc unregistered [ 4098.145441] LNet: 253801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4098.148626] LNetError: 253801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4098.157038] LNet: Removed LNI 192.168.203.119@tcp [ 4098.505152] Key type .llcrypt unregistered [ 4098.506804] Key type ._llcrypt unregistered [ 4109.071100] Key type ._llcrypt registered [ 4109.072405] Key type .llcrypt registered [ 4109.118433] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4109.598984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4109.611811] alg: No test for adler32 (adler32-zlib) [ 4110.503904] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 4110.597132] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 4112.199281] Key type lgssc registered [ 4112.683453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4115.958716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4115.967099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4117.177866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.712117] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4122.108582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4122.266314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4123.703668] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4125.765855] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4127.057630] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4129.686765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4129.820716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4131.863277] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4133.919451] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4142.114892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4157.414665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4157.415717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4157.423620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4157.426010] Lustre: Skipped 1 previous similar message [ 4159.891572] Lustre: server umount lustre-OST0000 complete [ 4162.528302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.530060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4162.533290] Lustre: Skipped 2 previous similar messages [ 4167.450205] Lustre: server umount lustre-MDT0000 complete [ 4167.648215] LustreError: 255176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4167.653572] LustreError: 255176:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4168.832131] LustreError: 255158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772761890 with bad export cookie 768873040708407421 [ 4168.835214] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4168.836858] LustreError: 255158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4168.954846] Lustre: server umount lustre-MDT0001 complete [ 4172.647104] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 4173.729410] Key type lgssc unregistered [ 4173.877074] LNet: 259144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4173.883849] LNetError: 259144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4173.891341] LNet: Removed LNI 192.168.203.119@tcp [ 4174.231555] Key type .llcrypt unregistered [ 4174.233191] Key type ._llcrypt unregistered [ 4181.722276] Key type ._llcrypt registered [ 4181.723552] Key type .llcrypt registered [ 4181.762107] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4182.231290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4182.241490] alg: No test for adler32 (adler32-zlib) [ 4183.103982] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 4183.191750] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 4184.775206] Key type lgssc registered [ 4185.179616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4188.152163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4188.159031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4189.380983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4190.888972] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4194.193412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4194.348785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4195.775765] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4197.839687] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4199.181645] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4201.676477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4201.818704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4203.852943] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4205.922609] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4216.168256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4220.383828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4220.386667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4220.392347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4221.407806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4221.410433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4221.411416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.293729] Lustre: server umount lustre-OST0000 complete [ 4224.992938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4224.995760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.000250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4225.002693] Lustre: Skipped 1 previous similar message [ 4225.817442] Lustre: server umount lustre-MDT0000 complete [ 4226.528604] LustreError: 260347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4226.535703] LustreError: 260347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4227.230249] LustreError: 262686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772761948 with bad export cookie 1560867945179381257 [ 4227.232735] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4227.235785] LustreError: 262686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4227.368122] Lustre: server umount lustre-MDT0001 complete [ 4231.176509] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 4232.232076] Key type lgssc unregistered [ 4232.368408] LNet: 263708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4232.372384] LNetError: 263708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4232.381638] LNet: Removed LNI 192.168.203.119@tcp [ 4232.685434] Key type .llcrypt unregistered [ 4232.686688] Key type ._llcrypt unregistered [ 4236.369454] Key type ._llcrypt registered [ 4236.370623] Key type .llcrypt registered [ 4236.409670] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4242.494831] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 4249.269923] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4249.777839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4249.792740] alg: No test for adler32 (adler32-zlib) [ 4250.655089] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 4250.739533] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 4252.327170] Key type lgssc registered [ 4252.690098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4264.888731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4264.898386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4266.009783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4266.030035] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4266.071193] Lustre: lustre-MDT0000: new disk, initializing [ 4266.103176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4266.112971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4267.607101] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4272.109685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4272.154017] Lustre: 267182:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4272.181699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4272.184181] Lustre: Skipped 1 previous similar message [ 4272.222201] Lustre: lustre-MDT0001: new disk, initializing [ 4272.261151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4272.272774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4272.276273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4273.712959] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4277.220029] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4278.650705] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4281.170621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4281.278323] Lustre: lustre-OST0000: new disk, initializing [ 4281.280944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4281.313880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.327126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4283.331347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4283.368124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4283.388135] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4286.890083] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4288.544377] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4288.638284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4290.198490] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4290.282105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4293.603387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4293.603864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.604513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4293.606046] LustreError: Skipped 1 previous similar message [ 4293.610706] Lustre: Skipped 1 previous similar message [ 4298.722238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4298.724968] Lustre: Skipped 2 previous similar messages [ 4303.843597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4303.845751] Lustre: Skipped 1 previous similar message [ 4305.375127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4305.454699] Lustre: server umount lustre-OST0000 complete [ 4308.447934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4308.452407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4308.458014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4314.080710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4314.083626] Lustre: Skipped 2 previous similar messages [ 4321.247107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4321.305305] Lustre: server umount lustre-MDT0000 complete [ 4322.790302] LustreError: 267175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772762044 with bad export cookie 7205058988836734244 [ 4322.792383] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4322.795411] LustreError: 267175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4322.927736] Lustre: server umount lustre-MDT0001 complete [ 4326.185614] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:54:07 (1772762047) [ 4328.771183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4328.958340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4330.427665] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4333.908278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4334.048217] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4334.050902] Lustre: Skipped 2 previous similar messages [ 4334.089768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4335.587080] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4337.716360] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4339.128236] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4341.616878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4343.858755] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4345.789357] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4360.161724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.162493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4360.166873] Lustre: Skipped 1 previous similar message [ 4360.172486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4360.175032] Lustre: Skipped 1 previous similar message [ 4365.598908] Lustre: server umount lustre-OST0000 complete [ 4369.888106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4369.889856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4369.891795] LustreError: Skipped 1 previous similar message [ 4369.899256] Lustre: Skipped 1 previous similar message [ 4373.337923] Lustre: server umount lustre-MDT0000 complete [ 4374.931480] LustreError: 271167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772762096 with bad export cookie 7205058988836735112 [ 4374.935148] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4374.935718] LustreError: 271905:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4374.935930] LustreError: 271905:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bfa5dbd4 x1858875657616128/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4374.936780] LustreError: 271167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4375.089081] Lustre: server umount lustre-MDT0001 complete [ 4379.218667] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 4380.363392] Key type lgssc unregistered [ 4380.507345] LNet: 274391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4380.510624] LNetError: 274391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4380.520330] LNet: Removed LNI 192.168.203.119@tcp [ 4380.870091] Key type .llcrypt unregistered [ 4380.870923] Key type ._llcrypt unregistered [ 4391.328921] Key type ._llcrypt registered [ 4391.330098] Key type .llcrypt registered [ 4391.372784] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 4398.201788] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4398.559800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4398.620371] alg: No test for adler32 (adler32-zlib) [ 4399.500571] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 4399.599301] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 4401.193451] Key type lgssc registered [ 4401.690189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4419.819552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4419.831949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4420.947693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4420.961024] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4421.003055] Lustre: lustre-MDT0000: new disk, initializing [ 4421.030920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.038913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4422.426339] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4426.816689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4426.867426] Lustre: 278154:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4426.890299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4426.892766] Lustre: Skipped 1 previous similar message [ 4426.932928] Lustre: lustre-MDT0001: new disk, initializing [ 4426.969548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4426.985532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4426.988806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4428.391149] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4431.678163] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4433.078627] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4435.591795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4435.708691] Lustre: lustre-OST2710: new disk, initializing [ 4435.711176] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4435.739612] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4437.821838] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4441.297753] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4443.136094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4443.139912] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4443.185989] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4443.950229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4444.017030] Lustre: lustre-OST03e8: new disk, initializing [ 4444.019547] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4444.056876] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4446.154436] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4448.248772] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4449.289786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4449.294925] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4449.348118] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4452.186223] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4452.270732] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4453.833238] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4453.918039] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4455.435161] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4455.514730] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4457.138502] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4457.219824] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4459.489916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4459.491679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4459.493902] Lustre: Skipped 3 previous similar messages [ 4459.495684] Lustre: Skipped 2 previous similar messages [ 4464.548478] Lustre: server umount lustre-MDT0000 complete [ 4466.059038] LustreError: 279905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772762187 with bad export cookie 10747276763882565107 [ 4466.060280] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4466.063796] LustreError: 279905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4466.191987] Lustre: server umount lustre-MDT0001 complete [ 4477.956189] Lustre: server umount lustre-OST2710 complete [ 4489.629325] Lustre: server umount lustre-OST03e8 complete [ 4495.916704] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 4499.348328] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4512.286373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4512.378132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4512.388860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4512.430136] Lustre: lustre-MDT0000: new disk, initializing [ 4512.463865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4512.471639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4513.880165] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4518.244669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4518.290814] Lustre: 285873:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4518.390386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4519.786960] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4523.108821] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4524.390121] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4526.882863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4526.986768] Lustre: lustre-OST0000: new disk, initializing [ 4526.988673] Lustre: Skipped 1 previous similar message [ 4526.990996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4526.993758] Lustre: Skipped 2 previous similar messages [ 4527.021902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4527.023869] Lustre: Skipped 1 previous similar message [ 4527.427591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4527.430793] Lustre: Skipped 1 previous similar message [ 4527.433141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4527.464700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4529.075874] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4532.366165] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4533.887114] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4533.967710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4535.380066] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4535.452986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4537.824618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4537.825113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4537.827519] LustreError: Skipped 1 previous similar message [ 4537.828227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4537.833893] Lustre: Skipped 1 previous similar message [ 4537.835722] Lustre: Skipped 1 previous similar message [ 4542.226906] Lustre: server umount lustre-OST0000 complete [ 4543.968041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4543.971414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.977628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4548.064733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4548.065538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4549.657690] Lustre: server umount lustre-MDT0000 complete [ 4550.898196] LustreError: 285864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772762272 with bad export cookie 10747276763882566773 [ 4550.901131] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4550.902020] LustreError: 285864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4551.007812] Lustre: server umount lustre-MDT0001 complete [ 4553.864332] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:57:55 (1772762275) [ 4554.384467] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4554.998755] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:57:56 (1772762276) [ 4556.006069] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4565.732368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4565.840966] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4565.849492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4565.920701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.344348] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4570.520597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4570.558262] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4570.569454] Lustre: 291300:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4572.116523] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4574.116464] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4575.340615] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4588.686871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4588.768996] Lustre: Denying initial registration attempt from nid 192.168.203.119@tcp, specified as failover [ 4588.775346] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4588.778572] LustreError: 292914:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -99 [ 4588.781912] LustreError: 292914:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4588.784142] LustreError: 292914:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4588.810165] Lustre: server umount lustre-OST0000 complete [ 4588.812210] LustreError: 292914:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4591.071831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4591.074703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4591.079830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4591.081892] Lustre: Skipped 1 previous similar message [ 4597.143752] Lustre: server umount lustre-MDT0000 complete [ 4598.409089] LustreError: 291293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772762320 with bad export cookie 10747276763882567641 [ 4598.411593] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4598.413573] LustreError: 291293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4598.528211] Lustre: server umount lustre-MDT0001 complete [ 4602.348732] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 4603.430148] Key type lgssc unregistered [ 4603.559399] LNet: 294017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4603.562955] LNetError: 294017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4603.570341] LNet: Removed LNI 192.168.203.119@tcp [ 4603.896563] Key type .llcrypt unregistered [ 4603.898094] Key type ._llcrypt unregistered [ 4608.457505] Key type ._llcrypt registered [ 4608.458744] Key type .llcrypt registered [ 4608.509587] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4608.987045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4608.998622] alg: No test for adler32 (adler32-zlib) [ 4609.884126] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 4610.024569] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 4611.631156] Key type lgssc registered [ 4612.155527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4650.547832] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4654.453557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4654.466466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4655.588268] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4655.598579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4655.665928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4657.278900] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4660.948264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4661.010834] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4661.024525] Lustre: 296731:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4661.151672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4662.919171] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4665.316495] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4667.012141] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4681.188330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4681.290592] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4681.392242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4684.053648] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4686.573437] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4687.597191] Lustre: server umount lustre-OST0000 complete [ 4690.964768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.971920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4695.469969] Lustre: server umount lustre-MDT0000 complete [ 4697.146689] LustreError: 296724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772762418 with bad export cookie 4100439666798465820 [ 4697.149370] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4697.151091] LustreError: 296724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4697.304044] Lustre: server umount lustre-MDT0001 complete [ 4701.182509] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 4702.302441] Key type lgssc unregistered [ 4702.435473] LNet: 299938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4702.439420] LNetError: 299938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4702.450198] LNet: Removed LNI 192.168.203.119@tcp [ 4702.769347] Key type .llcrypt unregistered [ 4702.770749] Key type ._llcrypt unregistered [ 4712.807878] Key type ._llcrypt registered [ 4712.809318] Key type .llcrypt registered [ 4712.851061] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4713.269905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4713.276155] alg: No test for adler32 (adler32-zlib) [ 4714.175230] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 4714.273602] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 4715.871208] Key type lgssc registered [ 4716.316086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4719.792538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4719.803917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.069180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4722.991260] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4726.760420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4726.961869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4728.686470] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4731.103223] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4732.675248] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4735.716853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4735.898046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4738.447441] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4740.913567] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4751.330411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4751.334275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4751.336634] Lustre: Skipped 1 previous similar message [ 4751.345628] Lustre: Skipped 2 previous similar messages [ 4755.362571] Lustre: server umount lustre-MDT0000 complete [ 4756.450023] LustreError: 302446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4756.455521] LustreError: 302446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4756.902487] LustreError: 302928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772762478 with bad export cookie 795554834897332256 [ 4756.903863] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4756.908121] LustreError: 302928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4761.568203] LustreError: 301311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.568498] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4761.573215] LustreError: 301311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4761.577304] Lustre: Skipped 1 previous similar message [ 4761.582908] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4761.585726] Lustre: Skipped 1 previous similar message [ 4766.692890] LustreError: 301311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4766.696959] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4766.700460] LustreError: 301311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4766.705326] Lustre: Skipped 1 previous similar message [ 4771.295197] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4771.412248] Lustre: server umount lustre-MDT0001 complete [ 4774.465267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4778.159117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4782.049056] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb0a60e8456683ef [ 4782.052672] Lustre: MGC192.168.203.119@tcp: Connection restored to 0@lo (at 0@lo) [ 4782.166469] LustreError: 302916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4782.177018] LustreError: 302916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4782.204387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4782.228778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4783.752916] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4787.493515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4787.668139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4789.446875] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4791.779524] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4791.843202] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4796.900640] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4797.643189] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4799.073289] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4799.696818] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4802.019071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4802.019749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4802.023652] Lustre: Skipped 1 previous similar message [ 4802.027866] Lustre: Skipped 2 previous similar messages [ 4807.137496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4807.140362] Lustre: Skipped 1 previous similar message [ 4807.329809] Lustre: server umount lustre-OST0000 complete [ 4812.257086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.262437] Lustre: Skipped 1 previous similar message [ 4815.097246] Lustre: server umount lustre-MDT0000 complete [ 4816.804548] LustreError: 301297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772762538 with bad export cookie 795554834897339375 [ 4816.807581] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4816.810476] LustreError: 301297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4816.972429] Lustre: server umount lustre-MDT0001 complete [ 4821.343707] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 4822.571241] Key type lgssc unregistered [ 4822.722705] LNet: 307010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4822.725832] LNetError: 307010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4822.735526] LNet: Removed LNI 192.168.203.119@tcp [ 4823.088874] Key type .llcrypt unregistered [ 4823.090353] Key type ._llcrypt unregistered [ 4834.145930] Key type ._llcrypt registered [ 4834.147488] Key type .llcrypt registered [ 4834.207742] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4834.601870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4834.633765] alg: No test for adler32 (adler32-zlib) [ 4835.516383] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 4835.644385] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 4837.239157] Key type lgssc registered [ 4837.872750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4841.533515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4841.547544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4842.758914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4844.324918] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4847.996698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4848.178795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4849.780451] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4851.977214] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4853.490293] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4857.760633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4857.761682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4857.765421] Lustre: Skipped 1 previous similar message [ 4860.824241] Lustre: server umount lustre-MDT0000 complete [ 4862.302196] LustreError: 309787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772762584 with bad export cookie 7925618320570935306 [ 4862.305636] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4862.306591] LustreError: 309787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4862.429796] Lustre: server umount lustre-MDT0001 complete [ 4868.698778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4868.829515] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4868.846859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4868.958682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4870.678900] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4874.223367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4874.286575] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4874.302195] Lustre: 310758:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4874.432116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4876.140484] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4878.294325] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4879.714495] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4882.760699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4882.864986] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4882.868533] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4882.872935] LustreError: 312413:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 4882.878510] LustreError: 312413:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4882.881050] LustreError: 312413:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4882.913050] Lustre: server umount lustre-OST0000 complete [ 4882.914813] LustreError: 312413:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4886.738345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4886.863485] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4886.959801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4889.303523] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4891.485757] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4894.711869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4894.748535] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4894.778583] Lustre: lustre-OST0001: new disk, initializing [ 4894.781400] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4896.038290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4896.041331] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4896.073772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4896.104412] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4896.969149] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4899.071811] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4901.346812] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4901.350044] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.354154] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4901.355557] Lustre: Skipped 1 previous similar message [ 4901.357980] Lustre: Skipped 1 previous similar message [ 4902.330251] Lustre: server umount lustre-OST0001 complete [ 4906.469399] LustreError: 312878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4906.470333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4906.470538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4906.470543] LustreError: Skipped 1 previous similar message [ 4906.477675] LustreError: 312878:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4906.482814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4906.491411] Lustre: Skipped 2 previous similar messages [ 4911.593709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4911.593891] LustreError: 312876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4911.598870] Lustre: Skipped 1 previous similar message [ 4911.609154] LustreError: 312876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4916.708808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4916.709056] LustreError: 314022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4916.712157] Lustre: Skipped 1 previous similar message [ 4916.725439] LustreError: 314022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4918.239115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4918.300859] Lustre: server umount lustre-OST0000 complete [ 4920.799796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4920.802719] LustreError: Skipped 1 previous similar message [ 4920.804358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.809599] Lustre: Skipped 1 previous similar message [ 4926.944586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4926.947688] Lustre: Skipped 4 previous similar messages [ 4934.111096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4934.190925] Lustre: server umount lustre-MDT0000 complete [ 4935.787677] LustreError: 310750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772762657 with bad export cookie 7925618320570935845 [ 4935.789869] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4935.799844] LustreError: 310750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4935.974939] Lustre: server umount lustre-MDT0001 complete [ 4940.640281] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 4941.870584] Key type lgssc unregistered [ 4942.028371] LNet: 315664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4942.032224] LNetError: 315664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4942.039772] LNet: Removed LNI 192.168.203.119@tcp [ 4942.397614] Key type .llcrypt unregistered [ 4942.400448] Key type ._llcrypt unregistered [ 4977.189439] Key type ._llcrypt registered [ 4977.191280] Key type .llcrypt registered [ 4977.233097] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 4983.945644] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4984.418587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4984.426409] alg: No test for adler32 (adler32-zlib) [ 4985.309648] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 4985.402240] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 4986.991234] Key type lgssc registered [ 4987.444533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5001.591505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5001.599841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5002.710298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5002.728859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5002.771890] Lustre: lustre-MDT0000: new disk, initializing [ 5002.800406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5002.808986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5004.437294] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5008.994519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5009.043361] Lustre: 320534:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5009.079926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5009.083412] Lustre: Skipped 1 previous similar message [ 5009.127220] Lustre: lustre-MDT0001: new disk, initializing [ 5009.172583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5009.187341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5009.192126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5010.665945] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5014.174711] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5015.680238] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5018.291313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5018.421942] Lustre: lustre-OST0000: new disk, initializing [ 5018.424788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5018.461741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5020.233119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5020.237762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5020.277060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5020.605047] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5023.939394] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5025.560740] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5025.662157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5027.261778] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5027.355031] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5030.880954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5030.882975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5030.886763] LustreError: Skipped 1 previous similar message [ 5030.896643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5030.899186] Lustre: Skipped 1 previous similar message [ 5034.288971] Lustre: server umount lustre-OST0000 complete [ 5036.000322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.001347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5036.004111] Lustre: Skipped 2 previous similar messages [ 5041.120689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5041.128707] Lustre: Skipped 2 previous similar messages [ 5046.240071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5046.243077] Lustre: Skipped 1 previous similar message [ 5050.335122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5050.336436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5050.341148] Lustre: Skipped 1 previous similar message [ 5050.405514] Lustre: server umount lustre-MDT0000 complete [ 5051.663127] LustreError: 320526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772762773 with bad export cookie 1796618335049337251 [ 5051.666733] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5051.667426] LustreError: 320526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5051.788794] Lustre: server umount lustre-MDT0001 complete [ 5054.831242] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:06:15 (1772762775) [ 5057.502728] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:06:18 (1772762778) [ 5060.673776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5060.862335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5062.228797] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5065.284245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5065.413612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5066.865147] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5068.727564] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5069.984715] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5072.343107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5074.474702] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5076.409645] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5090.695502] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5091.336786] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5091.916413] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5092.493520] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5093.143576] Lustre: Failing over lustre-MDT0000 [ 5093.305240] Lustre: server umount lustre-MDT0000 complete [ 5096.417866] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.418667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5096.419115] LustreError: 324876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5096.419126] LustreError: 324876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5096.422081] Lustre: Skipped 2 previous similar messages [ 5097.529914] LustreError: 324870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5097.534798] LustreError: 324870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5101.536913] LustreError: 324875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.542221] LustreError: 324875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5106.242585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5106.288559] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5106.379108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5106.382363] Lustre: Skipped 1 previous similar message [ 5106.404070] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5107.688652] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5107.764510] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5111.784816] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5111.798617] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5111.815343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5112.997575] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5113.532328] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5115.426546] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5116.111993] Lustre: Failing over lustre-MDT0000 [ 5116.246724] Lustre: server umount lustre-MDT0000 complete [ 5116.896525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5116.897967] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.900864] LustreError: 324875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5116.903940] Lustre: Skipped 1 previous similar message [ 5116.911470] LustreError: 324875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5122.016486] LustreError: 327025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5122.022210] LustreError: 327025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5129.276159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5129.325965] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5129.422812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5129.449106] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5130.853301] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5133.364236] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5134.825750] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5134.827863] Lustre: Skipped 2 previous similar messages [ 5134.840813] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5134.858836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5136.049913] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5136.630352] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5139.937266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.937514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5139.939027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5139.942702] Lustre: Skipped 2 previous similar messages [ 5145.112531] Lustre: server umount lustre-MDT0000 complete [ 5146.415449] LustreError: 329442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772762868 with bad export cookie 1796618335049339932 [ 5146.418834] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5146.419450] LustreError: 329442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5146.542090] Lustre: server umount lustre-MDT0001 complete [ 5158.173392] Lustre: server umount lustre-OST0000 complete [ 5164.257316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.478306] LustreError: 330670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5164.488525] LustreError: 330670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5164.521268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.120868] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5169.512597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5171.085306] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5173.212651] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5174.622271] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5177.280334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5179.755087] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5181.999344] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5189.732706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5195.235596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5195.237098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5195.238438] LustreError: Skipped 1 previous similar message [ 5195.239121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5195.244084] Lustre: Skipped 2 previous similar messages [ 5195.245682] Lustre: Skipped 5 previous similar messages [ 5196.182547] Lustre: server umount lustre-OST0000 complete [ 5199.792524] Lustre: server umount lustre-MDT0000 complete [ 5200.354503] LustreError: 330666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5200.363965] LustreError: 330666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5201.252300] LustreError: 330651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772762922 with bad export cookie 1796618335049340653 [ 5201.255924] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5201.256792] LustreError: 330651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5201.384747] Lustre: server umount lustre-MDT0001 complete [ 5205.368951] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 5206.518058] Key type lgssc unregistered [ 5206.660884] LNet: 333867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5206.665569] LNetError: 333867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5206.679509] LNet: Removed LNI 192.168.203.119@tcp [ 5207.001800] Key type .llcrypt unregistered [ 5207.003488] Key type ._llcrypt unregistered [ 5217.543678] Key type ._llcrypt registered [ 5217.544819] Key type .llcrypt registered [ 5217.581224] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5218.002252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5218.016959] alg: No test for adler32 (adler32-zlib) [ 5218.879756] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 5218.967353] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 5220.551719] Key type lgssc registered [ 5221.000023] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5223.931535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5223.939357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5225.171154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5226.708638] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5230.076912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5230.227030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5231.566885] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5233.391566] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5234.651843] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5237.513104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5237.655072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5239.901371] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5242.142340] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5249.958679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5260.259717] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.260986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5260.264546] Lustre: Skipped 2 previous similar messages [ 5260.267286] Lustre: Skipped 1 previous similar message [ 5265.201443] Lustre: server umount lustre-MDT0000 complete [ 5265.376346] LustreError: 336551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5265.381672] LustreError: 336551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5266.777644] LustreError: 335325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772762988 with bad export cookie 13390095814505435257 [ 5266.780902] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5266.783128] LustreError: 335325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5266.926450] Lustre: server umount lustre-MDT0001 complete [ 5278.778282] Lustre: server umount lustre-OST0000 complete [ 5284.931118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5285.146397] LustreError: 338737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.180442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5287.001944] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5290.466635] LustreError: 338738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.707925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5290.880940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5292.491537] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5294.801383] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5296.319899] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5299.124908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5299.251097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5301.702853] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5304.089325] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5309.480034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5316.582356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5316.585456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5316.591366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5319.648822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5319.649485] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5319.650809] Lustre: Skipped 1 previous similar message [ 5319.655214] Lustre: Skipped 1 previous similar message [ 5322.461148] Lustre: server umount lustre-MDT0000 complete [ 5323.811317] LustreError: 338762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772763045 with bad export cookie 13390095814505436356 [ 5323.815459] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5323.816028] LustreError: 338762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5323.943582] Lustre: server umount lustre-MDT0001 complete [ 5335.723920] Lustre: server umount lustre-OST0000 complete [ 5341.317230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5341.513882] LustreError: 342309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5341.552932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5343.126615] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5346.249451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5347.927207] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5349.886524] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5351.337512] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5353.858224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5356.099486] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5358.041727] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5366.311806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5371.361454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5371.362804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5371.371938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5371.374434] Lustre: Skipped 2 previous similar messages [ 5375.838523] Lustre: server umount lustre-OST0000 complete [ 5381.601902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5381.602965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5381.606187] Lustre: Skipped 2 previous similar messages [ 5383.397763] Lustre: server umount lustre-MDT0000 complete [ 5384.765291] LustreError: 342289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772763106 with bad export cookie 13390095814505437364 [ 5384.768144] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5384.771967] LustreError: 342289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5384.895843] Lustre: server umount lustre-MDT0001 complete [ 5389.060314] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 5390.268445] Key type lgssc unregistered [ 5390.412517] LNet: 345503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5390.418485] LNetError: 345503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5390.425439] LNet: Removed LNI 192.168.203.119@tcp [ 5390.786799] Key type .llcrypt unregistered [ 5390.787919] Key type ._llcrypt unregistered [ 5402.344386] Key type ._llcrypt registered [ 5402.346094] Key type .llcrypt registered [ 5402.399762] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5402.774756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5402.793043] alg: No test for adler32 (adler32-zlib) [ 5403.679693] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 5403.782823] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 5405.383229] Key type lgssc registered [ 5405.832580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5408.944603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5408.954561] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5408.957894] LustreError: 346920:0:(osd_handler.c:8531:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5408.966360] LustreError: 346920:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5408.968812] LustreError: 346920:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5408.971306] LustreError: 346920:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5408.974488] LustreError: 346920:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5412.035482] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5412.038304] LustreError: 347317:0:(osd_handler.c:8531:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5412.051536] LustreError: 347317:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5412.054539] LustreError: 347317:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5412.057424] LustreError: 347317:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5412.060405] LustreError: 347317:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5418.903226] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 5419.941442] Key type lgssc unregistered [ 5420.082384] LNet: 348301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5420.085556] LNetError: 348301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5420.092373] LNet: Removed LNI 192.168.203.119@tcp [ 5420.378726] Key type .llcrypt unregistered [ 5420.380184] Key type ._llcrypt unregistered [ 5426.286959] Key type ._llcrypt registered [ 5426.287933] Key type .llcrypt registered [ 5426.322451] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 5432.423240] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5432.766713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5432.822628] alg: No test for adler32 (adler32-zlib) [ 5433.662106] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 5433.738412] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 5435.319118] Key type lgssc registered [ 5435.685171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5446.934986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5446.943350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5448.058690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5448.073126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5448.110695] Lustre: lustre-MDT0000: new disk, initializing [ 5448.136980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5448.144410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5449.455512] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5453.685785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5453.732553] Lustre: 351397:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5453.758142] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5453.760112] Lustre: Skipped 1 previous similar message [ 5453.802826] Lustre: lustre-MDT0001: new disk, initializing [ 5453.839977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5453.854061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5453.858400] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5455.192687] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5458.344600] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5459.526428] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5461.700051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5461.804236] Lustre: lustre-OST0000: new disk, initializing [ 5461.806788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5461.832837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5463.643522] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5463.798444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5463.801746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5463.833756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5466.711656] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5468.096142] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5468.174983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5469.497344] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5469.571474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5474.272442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5474.275051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5474.275742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5474.278643] Lustre: Skipped 1 previous similar message [ 5474.281290] Lustre: Skipped 1 previous similar message [ 5476.317499] Lustre: server umount lustre-OST0000 complete [ 5479.392634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5479.392843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.393491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5479.395881] LustreError: Skipped 1 previous similar message [ 5479.399835] Lustre: Skipped 1 previous similar message [ 5484.512921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5484.515572] Lustre: Skipped 2 previous similar messages [ 5489.632584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5489.634518] Lustre: Skipped 1 previous similar message [ 5492.191106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5492.265230] Lustre: server umount lustre-MDT0000 complete [ 5493.655204] LustreError: 351390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772763215 with bad export cookie 12398599630562545735 [ 5493.659066] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5493.660454] LustreError: 351390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5493.783263] Lustre: server umount lustre-MDT0001 complete [ 5496.746394] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5497.294582] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:13:38 (1772763218) [ 5499.593803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5499.784146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5501.120696] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5504.191582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5504.312264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5505.626310] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5507.525311] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5508.804323] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5511.021810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5513.061712] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5514.982428] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5517.451675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5517.517477] Lustre: lustre-OST0001: new disk, initializing [ 5517.519766] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5517.553827] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5517.556384] Lustre: Skipped 1 previous similar message [ 5519.641345] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5522.432913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5522.439785] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5522.477949] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5523.000206] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5527.520481] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5527.520702] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5527.524238] LustreError: Skipped 1 previous similar message [ 5527.524790] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5527.528738] Lustre: Skipped 1 previous similar message [ 5531.081292] Lustre: server umount lustre-OST0001 complete [ 5534.694412] LustreError: 357044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5534.699426] LustreError: 357044:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5539.811703] LustreError: 357042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5539.819022] LustreError: 357042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5544.929164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5544.930156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5544.932056] LustreError: 357043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5544.932655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5544.932660] Lustre: Skipped 4 previous similar messages [ 5544.936346] Lustre: Skipped 1 previous similar message [ 5544.948612] LustreError: 357043:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5549.405528] Lustre: server umount lustre-OST0000 complete [ 5555.168516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5555.173315] Lustre: Skipped 1 previous similar message [ 5557.018093] Lustre: server umount lustre-MDT0000 complete [ 5558.321149] LustreError: 356166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772763280 with bad export cookie 12398599630562546603 [ 5558.324673] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5558.325082] LustreError: 356166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5564.471181] Lustre: server umount lustre-MDT0001 complete [ 5568.321981] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 5569.492843] Key type lgssc unregistered [ 5569.625385] LNet: 359798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5569.628424] LNetError: 359798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5569.635310] LNet: Removed LNI 192.168.203.119@tcp [ 5570.026026] Key type .llcrypt unregistered [ 5570.027541] Key type ._llcrypt unregistered [ 5588.403309] Key type ._llcrypt registered [ 5588.404985] Key type .llcrypt registered [ 5588.456303] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5588.803732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5588.857551] alg: No test for adler32 (adler32-zlib) [ 5589.707065] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 5589.790289] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 5591.375124] Key type lgssc registered [ 5591.733074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5594.332971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5594.340776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5595.436407] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5595.444129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5595.503444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5596.852576] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5599.767380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5599.805131] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5599.816761] Lustre: 362557:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5599.913029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5601.176353] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5602.874579] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5604.065582] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5605.343955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5605.345940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.349699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5606.880311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5606.880738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.000109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5612.002346] Lustre: Skipped 2 previous similar messages [ 5617.120554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5617.122892] Lustre: Skipped 1 previous similar message [ 5619.167219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5619.234052] Lustre: server umount lustre-MDT0000 complete [ 5620.628534] LustreError: 362549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772763342 with bad export cookie 17039974199407129660 [ 5620.631935] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5620.633904] LustreError: 362549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5620.749086] Lustre: server umount lustre-MDT0001 complete [ 5624.217545] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:15:45 (1772763345) [ 5626.629185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5626.814319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5628.144248] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5631.183608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5631.313040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5632.663200] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5634.572455] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5635.869966] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5638.370625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5638.458406] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5638.501219] LustreError: 366430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5638.507158] LustreError: 366430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5638.550271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5640.567692] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5642.459052] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5651.469904] Lustre: Permanently deactivating lustre-OST0000 [ 5651.475049] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5652.460542] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5653.349765] LustreError: 367348:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5653.354800] LustreError: 367348:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5653.358256] LustreError: 367348:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5653.992357] LustreError: 367396:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5653.994623] LustreError: 367396:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5660.753207] Lustre: server umount lustre-OST0000 complete [ 5661.683906] LustreError: 367646:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5661.686241] LustreError: 367646:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5664.224614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5664.225112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5664.229749] Lustre: Skipped 1 previous similar message [ 5672.442471] Lustre: server umount lustre-MDT0000 complete [ 5673.702114] LustreError: 364822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772763395 with bad export cookie 17039974199407130227 [ 5673.705607] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5673.706379] LustreError: 364822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5684.256171] Lustre: server umount lustre-MDT0001 complete [ 5687.647153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5689.298853] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5690.438199] LustreError: 368785:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5692.981750] Lustre: 369125:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.119@tcp [ 5693.787548] Lustre: server umount MGS complete [ 5698.409777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.613057] LustreError: 369847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5698.619241] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5698.624523] Lustre: Skipped 1 previous similar message [ 5698.646932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5700.164272] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5703.457972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.557483] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5705.003465] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5706.952832] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5708.272291] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5708.961773] Lustre: Permanently reactivating lustre-OST0000 [ 5708.970212] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5708.973169] Lustre: Skipped 2 previous similar messages [ 5737.439163] LustreError: 371497:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5740.875818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5740.993634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5740.996397] Lustre: Skipped 1 previous similar message [ 5743.067614] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5745.079101] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5757.408281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5757.409995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5757.411053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5757.414109] Lustre: Skipped 1 previous similar message [ 5757.416552] Lustre: Skipped 3 previous similar messages [ 5758.938934] Lustre: server umount lustre-OST0000 complete [ 5762.391756] Lustre: server umount lustre-MDT0000 complete [ 5762.528826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.530210] LustreError: 373211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5762.534579] Lustre: Skipped 1 previous similar message [ 5762.544752] LustreError: 373211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5763.934545] LustreError: 369827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772763485 with bad export cookie 17039974199407131263 [ 5763.937415] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5763.941194] LustreError: 369827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5764.081804] Lustre: server umount lustre-MDT0001 complete [ 5768.430596] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 5769.669820] Key type lgssc unregistered [ 5769.806462] LNet: 374261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5769.809630] LNetError: 374261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5769.817487] LNet: Removed LNI 192.168.203.119@tcp [ 5770.157261] Key type .llcrypt unregistered [ 5770.158517] Key type ._llcrypt unregistered [ 5776.753083] Key type ._llcrypt registered [ 5776.754333] Key type .llcrypt registered [ 5776.791517] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 5783.380184] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5783.713892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5783.767731] alg: No test for adler32 (adler32-zlib) [ 5784.641779] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 5784.763969] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 5786.359204] Key type lgssc registered [ 5786.832985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5799.908818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5799.919664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.047955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5801.071382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5801.111183] Lustre: lustre-MDT0000: new disk, initializing [ 5801.140343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5801.148570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5802.581879] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5807.502831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5807.554666] Lustre: 377359:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5807.582197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5807.584677] Lustre: Skipped 1 previous similar message [ 5807.623260] Lustre: lustre-MDT0001: new disk, initializing [ 5807.660160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5807.672694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5807.676269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5809.060766] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5812.275682] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5813.563553] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5816.577348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5816.678629] Lustre: lustre-OST0000: new disk, initializing [ 5816.680535] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5816.719567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5819.022402] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5822.649529] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5823.492446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5823.496825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5823.545775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5824.241829] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5824.324720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5826.054087] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5826.155476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5828.576745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5828.578843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5828.580515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5828.583613] Lustre: Skipped 1 previous similar message [ 5828.587520] Lustre: Skipped 1 previous similar message [ 5833.048884] Lustre: server umount lustre-OST0000 complete [ 5838.304083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5838.306921] LustreError: Skipped 1 previous similar message [ 5838.308677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5838.314696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5840.817877] Lustre: server umount lustre-MDT0000 complete [ 5842.460460] LustreError: 377352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772763564 with bad export cookie 7070886559764390787 [ 5842.463733] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5842.468383] LustreError: 377352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5842.600494] Lustre: server umount lustre-MDT0001 complete [ 5845.808985] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5846.399792] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 21:19:27 (1772763567) [ 5848.845624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5849.030149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5850.468761] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5853.509057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.652805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5855.155276] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5857.612972] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5859.105724] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5861.865311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5862.006586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5864.199423] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5866.353208] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5867.103305] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5877.883896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5879.778322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5879.778814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5879.779670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5879.779675] Lustre: Skipped 1 previous similar message [ 5879.781771] Lustre: Skipped 2 previous similar messages [ 5879.788817] LustreError: Skipped 1 previous similar message [ 5884.897362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5884.899986] Lustre: Skipped 2 previous similar messages [ 5885.521578] Lustre: server umount lustre-OST0000 complete [ 5889.504054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5889.506970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5889.513034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5900.257189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5900.259414] Lustre: Skipped 5 previous similar messages [ 5901.279117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5901.344827] Lustre: server umount lustre-MDT0000 complete [ 5902.810538] LustreError: 381390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772763624 with bad export cookie 7070886559764391655 [ 5902.812730] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5902.817587] LustreError: 381390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5902.947714] Lustre: server umount lustre-MDT0001 complete [ 5906.978991] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 5908.137127] Key type lgssc unregistered [ 5908.264454] LNet: 384803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5908.267327] LNetError: 384803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5908.275346] LNet: Removed LNI 192.168.203.119@tcp [ 5908.610938] Key type .llcrypt unregistered [ 5908.612685] Key type ._llcrypt unregistered [ 5917.344432] Key type ._llcrypt registered [ 5917.345885] Key type .llcrypt registered [ 5917.387535] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 5917.490518] Key type .llcrypt unregistered [ 5917.491814] Key type ._llcrypt unregistered [ 5924.589182] Key type ._llcrypt registered [ 5924.590572] Key type .llcrypt registered [ 5924.634571] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5925.065911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5925.085561] alg: No test for adler32 (adler32-zlib) [ 5925.966349] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 5926.063565] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 5927.663152] Key type lgssc registered [ 5928.092285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5931.064834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5931.077595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5932.289817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5933.742087] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5937.228857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5937.382919] LustreError: 386950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5937.389539] LustreError: 386950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5937.390344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5939.449199] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5941.516095] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5942.498192] LustreError: 386955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5942.503844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5942.508585] LustreError: 386955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5944.180447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.317931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5945.727489] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5954.530798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5954.533049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5954.536409] Lustre: Skipped 1 previous similar message [ 5954.536758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5954.539421] Lustre: Skipped 1 previous similar message [ 5959.648963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5959.651461] Lustre: Skipped 1 previous similar message [ 5960.409596] Lustre: server umount lustre-OST0000 complete [ 5964.769145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5964.769580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5964.770198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5964.773856] Lustre: Skipped 1 previous similar message [ 5967.834483] Lustre: server umount lustre-MDT0000 complete [ 5969.110971] LustreError: 388571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772763690 with bad export cookie 14056409688096111092 [ 5969.114149] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5969.115602] LustreError: 388571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5969.237327] Lustre: server umount lustre-MDT0001 complete [ 5972.963517] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 5974.021198] Key type lgssc unregistered [ 5974.149529] LNet: 389939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5974.152388] LNetError: 389939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5974.161326] LNet: Removed LNI 192.168.203.119@tcp [ 5974.482948] Key type .llcrypt unregistered [ 5974.484198] Key type ._llcrypt unregistered [ 5984.187892] Key type ._llcrypt registered [ 5984.189235] Key type .llcrypt registered [ 5984.232487] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5984.604547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5984.640659] alg: No test for adler32 (adler32-zlib) [ 5985.505266] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 5985.594374] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 5987.191200] Key type lgssc registered [ 5987.560291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5990.209523] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5990.216986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6005.663362] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6010.783259] LustreError: 391289:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.119@tcp: failed processing log, type 4: rc = -110 [ 6037.407156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6039.465227] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6041.417214] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6043.967780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6044.214348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6044.257520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6045.685697] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6049.073361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6049.236262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6050.801481] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6053.019727] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6054.429538] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6059.488363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6059.488733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6059.491446] LustreError: Skipped 1 previous similar message [ 6059.496808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6059.500518] Lustre: Skipped 1 previous similar message [ 6061.537808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6061.540589] Lustre: Skipped 1 previous similar message [ 6063.074679] Lustre: server umount lustre-OST0000 complete [ 6064.609375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6064.613797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6064.618387] Lustre: Skipped 1 previous similar message [ 6064.620542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6066.657438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6066.657498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.782667] Lustre: server umount lustre-MDT0000 complete [ 6071.775958] LustreError: 392215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6071.784267] LustreError: 392215:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6072.462442] LustreError: 393480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772763794 with bad export cookie 17574634609536225279 [ 6072.464870] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6072.469384] LustreError: 393480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6072.608705] Lustre: server umount lustre-MDT0001 complete [ 6076.761726] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 6077.827956] Key type lgssc unregistered [ 6077.961384] LNet: 394501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6077.964471] LNetError: 394501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6077.973348] LNet: Removed LNI 192.168.203.119@tcp [ 6078.298372] Key type .llcrypt unregistered [ 6078.299503] Key type ._llcrypt unregistered [ 6088.643768] Key type ._llcrypt registered [ 6088.645272] Key type .llcrypt registered [ 6088.682099] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 6089.089410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6089.103587] alg: No test for adler32 (adler32-zlib) [ 6089.967547] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 6090.057642] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 6091.647102] Key type lgssc registered [ 6092.060125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6094.757452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6094.764234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6095.928053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6097.397682] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6100.325808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.473687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6101.931365] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6103.848750] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6105.169237] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6108.063418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6108.183253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6110.116589] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6112.097399] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6120.487283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6125.538516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6125.541499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6125.544731] Lustre: Skipped 2 previous similar messages [ 6125.547083] Lustre: Skipped 1 previous similar message [ 6127.683066] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 6127.686967] Lustre: Skipped 1 previous similar message [ 6129.760493] Lustre: server umount lustre-MDT0000 complete [ 6130.656407] LustreError: 398034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6130.665028] LustreError: 398034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6130.744672] LustreError: 395857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.19@tcp arrived at 1772763852 with bad export cookie 6873185771649294475 [ 6130.750713] LustreError: 395857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6131.493101] LustreError: 397492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772763853 with bad export cookie 6873185771649293824 [ 6131.499055] LustreError: 397492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6131.499155] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6131.680300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6131.685509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6131.693357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6135.776417] LustreError: 395877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6135.777248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6135.784298] LustreError: 395877:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6136.802551] LustreError: 395872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6136.813902] LustreError: 395872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6137.619461] Lustre: server umount lustre-OST0000 complete [ 6140.312520] Lustre: server umount lustre-MDT0001 complete [ 6144.208987] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 6145.322355] Key type lgssc unregistered [ 6145.455432] LNet: 399229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6145.458537] LNetError: 399229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6145.466451] LNet: Removed LNI 192.168.203.119@tcp [ 6145.801516] Key type .llcrypt unregistered [ 6145.803352] Key type ._llcrypt unregistered [ 6155.367010] Key type ._llcrypt registered [ 6155.368437] Key type .llcrypt registered [ 6155.407641] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 6155.851766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6155.867535] alg: No test for adler32 (adler32-zlib) [ 6156.716489] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 6156.799364] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 6158.383153] Key type lgssc registered [ 6158.778162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6161.442588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6161.449633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6162.604902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6163.893845] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6166.957806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.077967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6168.354022] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6170.180598] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6171.395698] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6173.778138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6173.903247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6176.014980] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6177.930388] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6188.262787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6192.607955] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6192.610347] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6192.615623] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6193.633570] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6193.634566] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6195.606174] Lustre: server umount lustre-MDT0001 complete [ 6198.752529] LustreError: 400606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6198.753271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6198.753343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6198.753849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6198.753852] Lustre: Skipped 2 previous similar messages [ 6198.758136] LustreError: 400606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6203.400854] Lustre: server umount lustre-OST0000 complete [ 6203.871906] LustreError: 400605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6204.651377] Lustre: server umount lustre-MDT0000 complete [ 6208.905377] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 6209.819168] Key type lgssc unregistered [ 6209.939401] LNet: 403960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6209.941893] LNetError: 403960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6209.949304] LNet: Removed LNI 192.168.203.119@tcp [ 6210.208022] Key type .llcrypt unregistered [ 6210.209532] Key type ._llcrypt unregistered [ 6218.023149] Key type ._llcrypt registered [ 6218.024410] Key type .llcrypt registered [ 6218.060351] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 6224.297937] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 6224.703630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6224.710613] alg: No test for adler32 (adler32-zlib) [ 6225.557816] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 6225.641675] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 6227.223137] Key type lgssc registered [ 6227.579021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6239.881692] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 6245.059491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6245.064475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6246.154705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6246.168076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6246.205326] Lustre: lustre-MDT0000: new disk, initializing [ 6246.231048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6246.239200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6247.546584] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6253.518915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6253.614207] Lustre: lustre-OST0000: new disk, initializing [ 6253.617172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6253.620066] Lustre: Skipped 1 previous similar message [ 6253.641981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6255.484274] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6255.877670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6255.881218] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6255.890767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6258.679840] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6264.354951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6264.405128] Lustre: 407893:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6264.435813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6264.473691] Lustre: lustre-MDT0001: new disk, initializing [ 6264.523389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6264.545201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6264.549599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6265.882412] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6269.520504] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6269.601399] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6270.958960] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6273.072734] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 6276.668566] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.19@tcp (stopping) [ 6277.089765] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6277.089873] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6277.097646] Lustre: Skipped 1 previous similar message [ 6280.865335] Lustre: server umount lustre-MDT0001 complete [ 6287.329453] LustreError: 409493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6287.335321] LustreError: 409493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6292.448919] LustreError: 408557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6292.454599] LustreError: 408557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6297.056073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6297.058370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.062634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6297.064589] Lustre: Skipped 2 previous similar messages [ 6297.568529] LustreError: 407906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6299.409058] Lustre: server umount lustre-OST0000 complete [ 6306.922887] Lustre: server umount lustre-MDT0000 complete [ 6311.394649] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 6312.381242] Key type lgssc unregistered [ 6312.499327] LNet: 412474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6312.501974] LNetError: 412474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6312.513313] LNet: Removed LNI 192.168.203.119@tcp [ 6312.805756] Key type .llcrypt unregistered [ 6312.807334] Key type ._llcrypt unregistered [ 6325.217418] Key type ._llcrypt registered [ 6325.218359] Key type .llcrypt registered [ 6325.254260] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 6325.348393] Key type .llcrypt unregistered [ 6325.349736] Key type ._llcrypt unregistered [ 6332.020931] Key type ._llcrypt registered [ 6332.022576] Key type .llcrypt registered [ 6332.061753] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 6332.402632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6332.438032] alg: No test for adler32 (adler32-zlib) [ 6333.289976] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 6333.371099] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 6334.951156] Key type lgssc registered [ 6335.293492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6344.256828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6344.263354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6345.361076] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6345.373400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6345.410135] Lustre: lustre-MDT0000: new disk, initializing [ 6345.434300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6345.441932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6346.758896] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6351.192171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6351.234423] Lustre: 415786:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6351.258649] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6351.260972] Lustre: Skipped 1 previous similar message [ 6351.299360] Lustre: lustre-MDT0001: new disk, initializing [ 6351.336123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6351.347970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6351.351541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6352.787329] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6355.947990] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6357.181099] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6359.532440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6359.633834] Lustre: lustre-OST0000: new disk, initializing [ 6359.636098] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6359.667645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6361.547036] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6364.690753] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6368.766913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6368.771121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6368.796188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6373.857020] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6373.859029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6373.862797] Lustre: Skipped 2 previous similar messages [ 6373.865457] Lustre: Skipped 1 previous similar message [ 6378.715936] Lustre: server umount lustre-MDT0000 complete [ 6380.146405] LustreError: 416565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772764101 with bad export cookie 16401856980596137037 [ 6380.149536] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6380.151211] LustreError: 416565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6380.269297] Lustre: server umount lustre-MDT0001 complete [ 6391.904364] Lustre: server umount lustre-OST0000 complete [ 6395.492552] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:28:36 (1772764116) [ 6398.129644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6398.283112] LustreError: 419444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6398.290418] LustreError: 419444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6398.307429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6399.645687] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6402.470309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6402.596736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6403.875371] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6405.746464] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6406.860192] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6420.151945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6420.230823] Lustre: Found index 0 for lustre-OST0000, updating log [ 6420.296166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6422.184728] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6423.398598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6424.100578] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6427.425308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6427.425588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6427.427873] Lustre: Skipped 1 previous similar message [ 6428.448657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6428.451334] Lustre: Skipped 1 previous similar message [ 6431.503351] Lustre: server umount lustre-OST0000 complete [ 6444.094976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6444.095765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6444.100722] Lustre: Skipped 1 previous similar message [ 6449.120170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6449.123423] Lustre: Skipped 2 previous similar messages [ 6450.262816] Lustre: server umount lustre-MDT0000 complete [ 6451.510709] LustreError: 421060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772764173 with bad export cookie 16401856980596138416 [ 6451.513287] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6451.514585] LustreError: 421060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6451.636645] Lustre: server umount lustre-MDT0001 complete [ 6454.406456] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:29:35 (1772764175) [ 6482.911218] LNet: 1 local NIs in recovery (showing 1): 192.168.203.119@tcp [ 7043.792197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.027638] LustreError: 425509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7044.080989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7046.582576] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7049.186535] LustreError: 425510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7051.678599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7054.297453] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7057.349685] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7059.364363] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7074.006776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7074.205315] Lustre: Found index 0 for lustre-OST0000, updating log [ 7077.860479] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7081.162780] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7084.006904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7084.007423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7084.019673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7084.587604] Lustre: server umount lustre-OST0000 complete [ 7087.716465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7087.724131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7087.731212] Lustre: Skipped 1 previous similar message [ 7089.827447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7089.830476] Lustre: Skipped 3 previous similar messages [ 7093.042624] Lustre: server umount lustre-MDT0000 complete [ 7093.925659] LustreError: 425506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7093.938851] LustreError: 425506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7095.265613] LNet: 1 local NIs in recovery (showing 1): 192.168.203.119@tcp [ 7095.267645] LustreError: 425491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772764816 with bad export cookie 16401856980596139473 [ 7095.269195] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7095.281124] LustreError: 425491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7095.469850] Lustre: server umount lustre-MDT0001 complete [ 7116.997362] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 7122.031427] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7141.721724] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:41:02 (1772764862) [ 7151.187450] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7152.887483] Key type lgssc unregistered [ 7153.070724] LNet: 431736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7153.074324] LNetError: 431736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7206.374739] LNet: Removed LNI 192.168.203.119@tcp [ 7206.784573] Key type .llcrypt unregistered [ 7206.786274] Key type ._llcrypt unregistered [ 7214.553565] Key type ._llcrypt registered [ 7214.554778] Key type .llcrypt registered [ 7214.607801] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7214.986879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7215.052232] alg: No test for adler32 (adler32-zlib) [ 7215.981493] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 7216.117104] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7217.727181] Key type lgssc registered [ 7218.185637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7219.906488] LNet: Added LNI 192.168.203.119@tcp100 [8/256/0/180] [ 7220.263176] LNet: Added LNI 192.168.203.119@tcp101 [8/256/0/180] [ 7220.627989] LNet: Added LNI 192.168.203.119@tcp102 [8/256/0/180] [ 7221.027887] LNet: Added LNI 192.168.203.119@tcp103 [8/256/0/180] [ 7221.413531] LNet: Added LNI 192.168.203.119@tcp104 [8/256/0/180] [ 7221.778888] LNet: Added LNI 192.168.203.119@tcp105 [8/256/0/180] [ 7222.151236] LNet: Added LNI 192.168.203.119@tcp106 [8/256/0/180] [ 7222.499661] LNet: Added LNI 192.168.203.119@tcp107 [8/256/0/180] [ 7222.838932] LNet: Added LNI 192.168.203.119@tcp108 [8/256/0/180] [ 7223.185361] LNet: Added LNI 192.168.203.119@tcp109 [8/256/0/180] [ 7223.532178] LNet: Added LNI 192.168.203.119@tcp110 [8/256/0/180] [ 7223.907248] LNet: Added LNI 192.168.203.119@tcp111 [8/256/0/180] [ 7224.277365] LNet: Added LNI 192.168.203.119@tcp112 [8/256/0/180] [ 7224.602101] LNet: Added LNI 192.168.203.119@tcp113 [8/256/0/180] [ 7224.935929] LNet: Added LNI 192.168.203.119@tcp114 [8/256/0/180] [ 7225.276472] LNet: Added LNI 192.168.203.119@tcp115 [8/256/0/180] [ 7225.611435] LNet: Added LNI 192.168.203.119@tcp116 [8/256/0/180] [ 7225.978906] LNet: Added LNI 192.168.203.119@tcp117 [8/256/0/180] [ 7226.394174] LNet: Added LNI 192.168.203.119@tcp118 [8/256/0/180] [ 7226.812103] LNet: Added LNI 192.168.203.119@tcp119 [8/256/0/180] [ 7227.201520] LNet: Added LNI 192.168.203.119@tcp120 [8/256/0/180] [ 7227.605382] LNet: Added LNI 192.168.203.119@tcp121 [8/256/0/180] [ 7228.023433] LNet: Added LNI 192.168.203.119@tcp122 [8/256/0/180] [ 7228.429716] LNet: Added LNI 192.168.203.119@tcp123 [8/256/0/180] [ 7228.801801] LNet: Added LNI 192.168.203.119@tcp124 [8/256/0/180] [ 7229.204121] LNet: Added LNI 192.168.203.119@tcp125 [8/256/0/180] [ 7229.679832] LNet: Added LNI 192.168.203.119@tcp126 [8/256/0/180] [ 7230.072959] LNet: Added LNI 192.168.203.119@tcp127 [8/256/0/180] [ 7230.488569] LNet: Added LNI 192.168.203.119@tcp128 [8/256/0/180] [ 7230.968573] LNet: Added LNI 192.168.203.119@tcp129 [8/256/0/180] [ 7231.457611] LNet: Added LNI 192.168.203.119@tcp130 [8/256/0/180] [ 7231.874715] LNet: Added LNI 192.168.203.119@tcp131 [8/256/0/180] [ 7232.284400] LNet: Added LNI 192.168.203.119@tcp132 [8/256/0/180] [ 7232.646965] LNet: Added LNI 192.168.203.119@tcp133 [8/256/0/180] [ 7233.003568] LNet: Added LNI 192.168.203.119@tcp134 [8/256/0/180] [ 7233.374263] LNet: Added LNI 192.168.203.119@tcp135 [8/256/0/180] [ 7236.321259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7236.329839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7237.400962] Lustre: 434715:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.119@tcp: can't add NID 192.168.203.119@tcp130: rc = -75 [ 7237.484463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7237.504546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7237.561800] Lustre: lustre-MDT0000: new disk, initializing [ 7237.617894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7237.632264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7239.291951] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7244.609379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7244.677872] Lustre: 434731:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7244.724288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7244.727075] Lustre: Skipped 1 previous similar message [ 7244.768326] Lustre: lustre-MDT0001: new disk, initializing [ 7244.784017] Lustre: 435517:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.119@tcp: can't add NID 192.168.203.119@tcp130: rc = -75 [ 7244.788956] Lustre: 435517:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7244.852921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7244.874995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7244.884117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7246.098898] Lustre: 435648:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.119@tcp: can't add NID 192.168.203.119@tcp130: rc = -75 [ 7246.103901] Lustre: 435648:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7246.818337] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7250.701469] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7252.507451] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7255.820225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7255.998595] Lustre: lustre-OST0000: new disk, initializing [ 7256.001879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7256.054603] Lustre: 436485:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.119@tcp: can't add NID 192.168.203.119@tcp130: rc = -75 [ 7256.058054] Lustre: 436485:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7256.085733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7258.767581] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7262.540825] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7264.761122] Lustre: 437228:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.119@tcp: can't add NID 192.168.203.119@tcp130: rc = -75 [ 7264.764959] Lustre: 437228:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7264.792957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7264.797276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7264.856205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7269.861096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7269.861404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7269.870166] Lustre: Skipped 1 previous similar message [ 7269.870343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7269.874536] Lustre: Skipped 1 previous similar message [ 7271.657810] Lustre: server umount lustre-OST0000 complete [ 7274.976757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.980628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7274.982562] Lustre: Skipped 1 previous similar message [ 7279.534212] Lustre: server umount lustre-MDT0000 complete [ 7281.312254] LustreError: 434723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772765003 with bad export cookie 17915244435692697591 [ 7281.315702] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7281.316329] LustreError: 434723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7281.480568] Lustre: server umount lustre-MDT0001 complete [ 7286.118419] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7287.421882] Key type lgssc unregistered [ 7287.608741] LNet: 438258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7287.613181] LNetError: 438258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7288.671753] LNet: Removed LNI 192.168.203.119@tcp [ 7288.673946] LNetError: 438258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7288.682987] LNet: Removed LNI 192.168.203.119@tcp100 [ 7289.695197] LNet: Removed LNI 192.168.203.119@tcp101 [ 7289.698180] LNetError: 438258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7289.705544] LNetError: 438258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7289.710103] LNet: Removed LNI 192.168.203.119@tcp102 [ 7290.719097] LNet: Removed LNI 192.168.203.119@tcp103 [ 7291.743224] LNet: Removed LNI 192.168.203.119@tcp104 [ 7291.745433] LNetError: 438258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7291.748536] LNetError: 438258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7291.752408] LNet: Removed LNI 192.168.203.119@tcp105 [ 7292.767182] LNet: Removed LNI 192.168.203.119@tcp106 [ 7293.791483] LNet: Removed LNI 192.168.203.119@tcp107 [ 7294.815150] LNet: Removed LNI 192.168.203.119@tcp108 [ 7295.840288] LNet: Removed LNI 192.168.203.119@tcp109 [ 7295.842123] LNetError: 438258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7295.844939] LNetError: 438258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7295.848071] LNet: Removed LNI 192.168.203.119@tcp110 [ 7296.863227] LNet: Removed LNI 192.168.203.119@tcp111 [ 7297.887397] LNet: Removed LNI 192.168.203.119@tcp112 [ 7297.889373] LNet: Removed LNI 192.168.203.119@tcp113 [ 7297.891131] LNet: Removed LNI 192.168.203.119@tcp114 [ 7297.892809] LNet: Removed LNI 192.168.203.119@tcp115 [ 7297.894254] LNet: Removed LNI 192.168.203.119@tcp116 [ 7297.895570] LNet: Removed LNI 192.168.203.119@tcp117 [ 7297.896680] LNet: Removed LNI 192.168.203.119@tcp118 [ 7297.898150] LNet: Removed LNI 192.168.203.119@tcp119 [ 7297.899664] LNet: Removed LNI 192.168.203.119@tcp120 [ 7297.900789] LNet: Removed LNI 192.168.203.119@tcp121 [ 7297.902050] LNet: Removed LNI 192.168.203.119@tcp122 [ 7297.903243] LNet: Removed LNI 192.168.203.119@tcp123 [ 7297.904756] LNet: Removed LNI 192.168.203.119@tcp124 [ 7297.905971] LNet: Removed LNI 192.168.203.119@tcp125 [ 7297.907595] LNet: Removed LNI 192.168.203.119@tcp126 [ 7297.909129] LNet: Removed LNI 192.168.203.119@tcp127 [ 7297.910562] LNet: Removed LNI 192.168.203.119@tcp128 [ 7297.912155] LNet: Removed LNI 192.168.203.119@tcp129 [ 7297.913627] LNet: Removed LNI 192.168.203.119@tcp130 [ 7297.915255] LNet: Removed LNI 192.168.203.119@tcp131 [ 7297.917084] LNet: Removed LNI 192.168.203.119@tcp132 [ 7297.918787] LNet: Removed LNI 192.168.203.119@tcp133 [ 7297.920387] LNet: Removed LNI 192.168.203.119@tcp134 [ 7297.927507] LNet: Removed LNI 192.168.203.119@tcp135 [ 7298.257693] Key type .llcrypt unregistered [ 7298.260226] Key type ._llcrypt unregistered [ 7307.756218] Key type ._llcrypt registered [ 7307.757543] Key type .llcrypt registered [ 7307.805943] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 7315.257228] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7315.719477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7315.733758] alg: No test for adler32 (adler32-zlib) [ 7316.614544] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 7316.714352] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7318.303564] Key type lgssc registered [ 7318.806451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7335.185853] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 21:44:16 (1772765056) [ 7343.050225] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7344.349583] Key type lgssc unregistered [ 7344.501857] LNet: 442265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7344.505393] LNetError: 442265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7344.514260] LNet: Removed LNI 192.168.203.119@tcp [ 7344.907729] Key type .llcrypt unregistered [ 7344.909085] Key type ._llcrypt unregistered [ 7353.338941] Key type ._llcrypt registered [ 7353.340442] Key type .llcrypt registered [ 7353.393315] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7353.789787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7353.798910] alg: No test for adler32 (adler32-zlib) [ 7354.698263] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 7354.816960] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7356.410067] Key type lgssc registered [ 7356.953225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7361.649375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7361.661571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7362.780659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7362.800526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7362.847772] Lustre: lustre-MDT0000: new disk, initializing [ 7362.881972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7362.891015] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7364.494782] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7369.631465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7369.711432] Lustre: 443437:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7369.741535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7369.744639] Lustre: Skipped 1 previous similar message [ 7369.786304] Lustre: lustre-MDT0001: new disk, initializing [ 7369.836907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7369.862156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7369.874765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7371.761728] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7375.651531] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7377.264447] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7380.315130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7380.430445] Lustre: lustre-OST0000: new disk, initializing [ 7380.432912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7380.475437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7383.048690] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7384.130480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7384.135702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7384.172713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7387.072341] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7388.703991] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing lnet_if_list [ 7390.872164] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7390.886759] LNet: Added LNI 192.168.203.119@tcp10 [8/256/0/180] [ 7393.415225] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7393.432018] LNet: Added LNI 192.168.203.119@tcp11 [8/256/0/180] [ 7395.250961] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7395.267309] LNet: Added LNI 192.168.203.119@tcp12 [8/256/0/180] [ 7396.937964] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7396.954166] LNet: Added LNI 192.168.203.119@tcp13 [8/256/0/180] [ 7398.639031] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7398.652652] LNet: Added LNI 192.168.203.119@tcp14 [8/256/0/180] [ 7400.763687] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7400.777149] LNet: Added LNI 192.168.203.119@tcp15 [8/256/0/180] [ 7402.649331] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7402.663583] LNet: Added LNI 192.168.203.119@tcp16 [8/256/0/180] [ 7404.831315] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7404.846921] LNet: Added LNI 192.168.203.119@tcp17 [8/256/0/180] [ 7406.926076] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7406.941346] LNet: Added LNI 192.168.203.119@tcp18 [8/256/0/180] [ 7409.010603] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7409.027484] LNet: Added LNI 192.168.203.119@tcp19 [8/256/0/180] [ 7411.123893] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7411.140638] LNet: Added LNI 192.168.203.119@tcp20 [8/256/0/180] [ 7413.288723] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7413.310243] LNet: Added LNI 192.168.203.119@tcp21 [8/256/0/180] [ 7415.490798] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7415.505478] LNet: Added LNI 192.168.203.119@tcp22 [8/256/0/180] [ 7418.006739] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7418.020711] LNet: Added LNI 192.168.203.119@tcp23 [8/256/0/180] [ 7420.318684] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7420.332186] LNet: Added LNI 192.168.203.119@tcp24 [8/256/0/180] [ 7422.823966] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7422.839593] LNet: Added LNI 192.168.203.119@tcp25 [8/256/0/180] [ 7425.255287] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7425.272932] LNet: Added LNI 192.168.203.119@tcp26 [8/256/0/180] [ 7427.671723] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7427.686209] LNet: Added LNI 192.168.203.119@tcp27 [8/256/0/180] [ 7430.058573] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7430.072445] LNet: Added LNI 192.168.203.119@tcp28 [8/256/0/180] [ 7432.625951] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7432.640761] LNet: Added LNI 192.168.203.119@tcp29 [8/256/0/180] [ 7435.210693] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7435.223931] LNet: Added LNI 192.168.203.119@tcp30 [8/256/0/180] [ 7437.558332] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7437.574191] LNet: Added LNI 192.168.203.119@tcp31 [8/256/0/180] [ 7440.475041] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7440.491758] LNet: Added LNI 192.168.203.119@tcp32 [8/256/0/180] [ 7443.315732] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7443.331864] LNet: Added LNI 192.168.203.119@tcp33 [8/256/0/180] [ 7446.231640] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7446.245878] LNet: Added LNI 192.168.203.119@tcp34 [8/256/0/180] [ 7448.803187] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7448.817921] LNet: Added LNI 192.168.203.119@tcp35 [8/256/0/180] [ 7451.486137] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7451.498928] LNet: Added LNI 192.168.203.119@tcp36 [8/256/0/180] [ 7454.368921] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7454.387846] LNet: Added LNI 192.168.203.119@tcp37 [8/256/0/180] [ 7457.344174] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7457.360844] LNet: Added LNI 192.168.203.119@tcp38 [8/256/0/180] [ 7460.365033] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7460.378320] LNet: Added LNI 192.168.203.119@tcp39 [8/256/0/180] [ 7463.215439] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7463.234207] LNet: Added LNI 192.168.203.119@tcp40 [8/256/0/180] [ 7466.327711] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7466.344786] LNet: Added LNI 192.168.203.119@tcp41 [8/256/0/180] [ 7469.289583] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7469.303834] LNet: Added LNI 192.168.203.119@tcp42 [8/256/0/180] [ 7472.711526] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7472.731339] LNet: Added LNI 192.168.203.119@tcp43 [8/256/0/180] [ 7475.912309] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7475.926463] LNet: Added LNI 192.168.203.119@tcp44 [8/256/0/180] [ 7479.307106] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7479.324801] LNet: Added LNI 192.168.203.119@tcp45 [8/256/0/180] [ 7482.607337] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7482.623833] LNet: Added LNI 192.168.203.119@tcp46 [8/256/0/180] [ 7485.754609] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7485.768758] LNet: Added LNI 192.168.203.119@tcp47 [8/256/0/180] [ 7489.244339] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7489.264708] LNet: Added LNI 192.168.203.119@tcp48 [8/256/0/180] [ 7493.261402] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7493.284442] LNet: Added LNI 192.168.203.119@tcp49 [8/256/0/180] [ 7497.646736] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7497.663137] LNet: Added LNI 192.168.203.119@tcp50 [8/256/0/180] [ 7510.269317] Lustre: server umount lustre-MDT0000 complete [ 7512.018774] LustreError: 443428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772765233 with bad export cookie 15323614029427633415 [ 7512.026081] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7512.033200] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7512.039254] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7512.176594] Lustre: server umount lustre-MDT0001 complete [ 7521.572563] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7525.996247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7537.632164] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd4a869c59339f588 [ 7537.634755] Lustre: MGC192.168.203.119@tcp: Connection restored to 0@lo (at 0@lo) [ 7537.769737] LustreError: 445168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7537.779205] LustreError: 445168:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7537.815311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7539.897258] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7543.267458] LustreError: 445169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7544.216937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7544.389076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7546.423074] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7549.491321] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7551.348377] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7551.464886] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7563.744346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7563.747536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7563.759402] Lustre: Skipped 1 previous similar message [ 7563.762052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7564.768218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7564.778424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7564.788134] LustreError: 445624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7564.853448] Lustre: server umount lustre-OST0000 complete [ 7569.887974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7569.892558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7569.901917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7571.936695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7573.632497] Lustre: server umount lustre-MDT0000 complete [ 7575.633239] LustreError: 443430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772765297 with bad export cookie 15323614029427635592 [ 7575.635962] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7575.640209] LustreError: 443430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7575.794687] Lustre: server umount lustre-MDT0001 complete [ 7580.846142] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7582.130466] Key type lgssc unregistered [ 7582.323051] LNet: 471118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7582.330264] LNetError: 471118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7582.335153] LNet: Removed LNI 192.168.203.119@tcp [ 7582.337314] LNet: Removed LNI 192.168.203.119@tcp10 [ 7582.341082] LNet: Removed LNI 192.168.203.119@tcp11 [ 7582.343222] LNet: Removed LNI 192.168.203.119@tcp12 [ 7582.345552] LNet: Removed LNI 192.168.203.119@tcp13 [ 7582.347976] LNet: Removed LNI 192.168.203.119@tcp14 [ 7582.350180] LNet: Removed LNI 192.168.203.119@tcp15 [ 7582.351420] LNet: Removed LNI 192.168.203.119@tcp16 [ 7582.355091] LNet: Removed LNI 192.168.203.119@tcp17 [ 7582.359318] LNet: Removed LNI 192.168.203.119@tcp18 [ 7582.360536] LNet: Removed LNI 192.168.203.119@tcp19 [ 7582.363627] LNet: Removed LNI 192.168.203.119@tcp20 [ 7582.367209] LNet: Removed LNI 192.168.203.119@tcp21 [ 7582.370192] LNet: Removed LNI 192.168.203.119@tcp22 [ 7582.372818] LNet: Removed LNI 192.168.203.119@tcp23 [ 7582.376136] LNet: Removed LNI 192.168.203.119@tcp24 [ 7582.379754] LNet: Removed LNI 192.168.203.119@tcp25 [ 7582.381419] LNet: Removed LNI 192.168.203.119@tcp26 [ 7582.385737] LNet: Removed LNI 192.168.203.119@tcp27 [ 7582.388468] LNet: Removed LNI 192.168.203.119@tcp28 [ 7582.390475] LNet: Removed LNI 192.168.203.119@tcp29 [ 7582.392774] LNet: Removed LNI 192.168.203.119@tcp30 [ 7582.394607] LNet: Removed LNI 192.168.203.119@tcp31 [ 7582.396236] LNet: Removed LNI 192.168.203.119@tcp32 [ 7582.397521] LNet: Removed LNI 192.168.203.119@tcp33 [ 7582.398963] LNet: Removed LNI 192.168.203.119@tcp34 [ 7582.400636] LNet: Removed LNI 192.168.203.119@tcp35 [ 7582.402266] LNet: Removed LNI 192.168.203.119@tcp36 [ 7582.403800] LNet: Removed LNI 192.168.203.119@tcp37 [ 7582.405756] LNet: Removed LNI 192.168.203.119@tcp38 [ 7582.407099] LNet: Removed LNI 192.168.203.119@tcp39 [ 7582.408274] LNet: Removed LNI 192.168.203.119@tcp40 [ 7582.409536] LNet: Removed LNI 192.168.203.119@tcp41 [ 7582.410659] LNet: Removed LNI 192.168.203.119@tcp42 [ 7582.412247] LNet: Removed LNI 192.168.203.119@tcp43 [ 7582.413993] LNet: Removed LNI 192.168.203.119@tcp44 [ 7582.415755] LNet: Removed LNI 192.168.203.119@tcp45 [ 7582.418151] LNet: Removed LNI 192.168.203.119@tcp46 [ 7582.419994] LNet: Removed LNI 192.168.203.119@tcp47 [ 7582.421312] LNet: Removed LNI 192.168.203.119@tcp48 [ 7582.422601] LNet: Removed LNI 192.168.203.119@tcp49 [ 7582.433340] LNet: Removed LNI 192.168.203.119@tcp50 [ 7582.735111] Key type .llcrypt unregistered [ 7582.737076] Key type ._llcrypt unregistered [ 7591.928479] Key type ._llcrypt registered [ 7591.929706] Key type .llcrypt registered [ 7591.977503] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7592.468621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7592.495453] alg: No test for adler32 (adler32-zlib) [ 7593.370917] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 7593.462525] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7595.055666] Key type lgssc registered [ 7595.523253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7598.919691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7598.929506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7600.178906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7601.904994] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7605.638429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7605.795104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7607.364522] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7609.780733] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7611.506626] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7614.580235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7614.766964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7617.390061] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7619.879550] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7624.998581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7630.475933] loop0: detected capacity change from 0 to 204800000 [ 7630.605195] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7630.609917] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7630.615985] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7633.027523] loop0: detected capacity change from 0 to 204800000 [ 7633.099791] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7633.104181] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7633.109926] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7635.249861] loop0: detected capacity change from 0 to 204800000 [ 7637.664554] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7637.738832] Lustre: fs15246-OST0000: new disk, initializing [ 7637.741581] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7637.777247] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7640.176680] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7644.883711] loop1: detected capacity change from 0 to 204800000 [ 7647.374528] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7647.424638] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7647.461842] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7647.517893] Lustre: fs15246-MDT0000: new disk, initializing [ 7647.569438] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7647.579577] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7647.622934] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7648.672133] Lustre: 476707:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000034332b08 x1858879162975104/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1772765386 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7649.299423] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7657.761908] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7657.765551] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7657.772857] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7657.962362] Lustre: server umount fs15246-MDT0000 complete [ 7660.006834] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7660.012109] LustreError: 472345:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7660.018569] LustreError: 472345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7667.774584] Lustre: server umount fs15246-OST0000 complete [ 7673.312836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7673.315697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7673.319414] Lustre: Skipped 1 previous similar message [ 7678.429077] Lustre: server umount lustre-OST0000 complete [ 7682.529085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7682.531956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.538384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7682.541528] Lustre: Skipped 1 previous similar message [ 7683.552380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7683.555431] Lustre: Skipped 1 previous similar message [ 7688.672317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7688.677304] Lustre: Skipped 1 previous similar message [ 7693.793784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7693.796461] Lustre: Skipped 1 previous similar message [ 7694.303178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7694.371279] Lustre: server umount lustre-MDT0000 complete [ 7696.030423] LustreError: 473067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772765417 with bad export cookie 858529935206013164 [ 7696.030721] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7696.035227] LustreError: 473067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7696.184819] Lustre: server umount lustre-MDT0001 complete [ 7700.866348] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7702.118239] Key type lgssc unregistered [ 7702.266386] LNet: 479769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7702.269470] LNetError: 479769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7702.277410] LNet: Removed LNI 192.168.203.119@tcp [ 7702.640772] Key type .llcrypt unregistered [ 7702.642191] Key type ._llcrypt unregistered [ 7711.438741] Key type ._llcrypt registered [ 7711.440420] Key type .llcrypt registered [ 7711.500234] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7712.060622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7712.071274] alg: No test for adler32 (adler32-zlib) [ 7712.974648] Lustre: Lustre: Build Version: 2.17.50_193_gbc7a75b [ 7713.126642] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7714.751596] Key type lgssc registered [ 7715.339828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7732.293812] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 7736.292887] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7752.331417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7752.347857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7753.475101] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7753.495074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7753.541687] Lustre: lustre-MDT0000: new disk, initializing [ 7753.575608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7753.583794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7755.463485] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7761.348886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.398868] Lustre: 484156:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7761.421794] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7761.423488] Lustre: Skipped 1 previous similar message [ 7761.461228] Lustre: lustre-MDT0001: new disk, initializing [ 7761.516123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7761.539925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7761.547768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7763.271892] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7767.366851] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7769.016097] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7772.289529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7772.425329] Lustre: lustre-OST0000: new disk, initializing [ 7772.427822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7772.472642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7775.209065] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7777.807031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7777.813982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7777.874460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7779.199117] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7781.039327] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7781.124340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7783.157735] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7783.270837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7788.001568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7788.009757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7788.015501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7788.017558] Lustre: Skipped 1 previous similar message [ 7790.311575] Lustre: server umount lustre-OST0000 complete [ 7792.100278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7792.102989] LustreError: Skipped 1 previous similar message [ 7792.106298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7792.112984] Lustre: Skipped 1 previous similar message [ 7792.115708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7796.195171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7796.196922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7798.251650] Lustre: server umount lustre-MDT0000 complete [ 7799.995623] LustreError: 484147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772765521 with bad export cookie 4205271586733611251 [ 7799.998319] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7800.000820] LustreError: 484147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7800.130197] Lustre: server umount lustre-MDT0001 complete [ 7811.660482] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 7815.769516] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7828.558300] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7693 sec ========== 21:52:29 (1772765549) [ 7829.196797] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:52:30 (1772765550) === [ 7830.542327] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:52:31 (1772765551) === [ 7845.179360] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7846.351122] Key type lgssc unregistered [ 7846.498286] LNet: 492327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7846.502890] LNetError: 492327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7846.514977] LNet: Removed LNI 192.168.203.119@tcp [ 7846.836571] Key type .llcrypt unregistered [ 7846.838332] Key type ._llcrypt unregistered