[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 457660174 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002282] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.006937] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008003] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009006] pid_max: default: 32768 minimum: 301 [ 0.010087] LSM: Security Framework initializing [ 0.011025] Yama: becoming mindful. [ 0.012018] SELinux: Initializing. [ 0.013036] *** VALIDATE selinux *** [ 0.020256] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024622] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026092] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027090] *** VALIDATE tmpfs *** [ 0.028306] *** VALIDATE proc *** [ 0.029217] *** VALIDATE cgroup *** [ 0.030005] *** VALIDATE cgroup2 *** [ 0.031219] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032140] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034022] Spectre V2 : User space: Vulnerable [ 0.035004] Speculative Store Bypass: Vulnerable [ 0.037510] debug: unmapping init [mem 0xffffffff92a59000-0xffffffff92a60fff] [ 0.040000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040934] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041025] ... version: 2 [ 0.042021] ... bit width: 48 [ 0.043009] ... generic registers: 4 [ 0.044009] ... value mask: 0000ffffffffffff [ 0.045011] ... max period: 00007fffffffffff [ 0.046012] ... fixed-purpose events: 3 [ 0.046929] ... event mask: 000000070000000f [ 0.047420] rcu: Hierarchical SRCU implementation. [ 0.049782] smp: Bringing up secondary CPUs ... [ 0.050756] x86: Booting SMP configuration: [ 0.051026] .... node #0, CPUs: #1 #2 #3 [ 0.054277] smp: Brought up 1 node, 4 CPUs [ 0.056012] smpboot: Max logical packages: 1 [ 0.057021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150372] node 0 deferred pages initialised in 92ms [ 0.153111] devtmpfs: initialized [ 0.155245] x86/mm: Memory block size: 128MB [ 0.159074] gcov: version magic: 0x41383552 [ 0.161182] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.164091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166235] pinctrl core: initialized pinctrl subsystem [ 0.168126] [ 0.168596] ************************************************************* [ 0.170011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172009] ** ** [ 0.174011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.176010] ** ** [ 0.178009] ** This means that this kernel is built to expose internal ** [ 0.180008] ** IOMMU data structures, which may compromise security on ** [ 0.182008] ** your system. ** [ 0.183009] ** ** [ 0.185018] ** If you see this message and you are not debugging the ** [ 0.187008] ** kernel, report this immediately to your vendor! ** [ 0.189007] ** ** [ 0.191010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192010] ************************************************************* [ 0.195558] NET: Registered protocol family 16 [ 0.199444] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.206455] cpuidle: using governor menu [ 0.207688] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.209548] PCI: Using configuration type 1 for base access [ 0.211202] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220244] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.222103] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224152] cryptd: max_cpu_qlen set to 1000 [ 0.226304] ACPI: Added _OSI(Module Device) [ 0.227015] ACPI: Added _OSI(Processor Device) [ 0.229010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233134] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.239413] ACPI: Interpreter enabled [ 0.240050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.241010] ACPI: Using IOAPIC for interrupt routing [ 0.242094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245374] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253513] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.255027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.257017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.260067] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.264095] acpiphp: Slot [2] registered [ 0.265046] acpiphp: Slot [5] registered [ 0.265782] acpiphp: Slot [6] registered [ 0.266079] acpiphp: Slot [7] registered [ 0.266938] acpiphp: Slot [8] registered [ 0.268079] acpiphp: Slot [9] registered [ 0.269064] acpiphp: Slot [10] registered [ 0.270011] acpiphp: Slot [3] registered [ 0.270940] acpiphp: Slot [4] registered [ 0.272046] acpiphp: Slot [11] registered [ 0.273066] acpiphp: Slot [12] registered [ 0.274056] acpiphp: Slot [13] registered [ 0.275070] acpiphp: Slot [14] registered [ 0.276062] acpiphp: Slot [15] registered [ 0.277052] acpiphp: Slot [16] registered [ 0.278066] acpiphp: Slot [17] registered [ 0.280066] acpiphp: Slot [18] registered [ 0.281101] acpiphp: Slot [19] registered [ 0.282074] acpiphp: Slot [20] registered [ 0.283067] acpiphp: Slot [21] registered [ 0.284062] acpiphp: Slot [22] registered [ 0.286065] acpiphp: Slot [23] registered [ 0.288055] acpiphp: Slot [24] registered [ 0.289063] acpiphp: Slot [25] registered [ 0.290064] acpiphp: Slot [26] registered [ 0.291096] acpiphp: Slot [27] registered [ 0.293073] acpiphp: Slot [28] registered [ 0.294069] acpiphp: Slot [29] registered [ 0.295067] acpiphp: Slot [30] registered [ 0.297079] acpiphp: Slot [31] registered [ 0.298034] PCI host bridge to bus 0000:00 [ 0.298890] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.300027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.302016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.304012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.306013] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.309021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.310163] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.312842] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.316000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.322957] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.326640] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.329015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.331016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.336241] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.337584] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.340031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.342721] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.350016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.364014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.368009] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.372661] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.378015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.385015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.400032] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.413155] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.419014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.424010] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.437011] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.446000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.453019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.458020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.470010] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.478582] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.486016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.491010] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.500014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.510934] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.515011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.520015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.532019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.541634] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.547013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.552014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.563014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.572222] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.575053] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.577305] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.580372] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.583186] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.588109] iommu: Default domain type: Passthrough [ 0.591370] SCSI subsystem initialized [ 0.592110] ACPI: bus type USB registered [ 0.594083] usbcore: registered new interface driver usbfs [ 0.596067] usbcore: registered new interface driver hub [ 0.597062] usbcore: registered new device driver usb [ 0.599124] pps_core: LinuxPPS API ver. 1 registered [ 0.601007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.604042] PTP clock support registered [ 0.606118] EDAC MC: Ver: 3.0.0 [ 0.608175] PCI: Using ACPI for IRQ routing [ 0.610208] NetLabel: Initializing [ 0.612013] NetLabel: domain hash size = 128 [ 0.613009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.615101] NetLabel: unlabeled traffic allowed by default [ 0.618147] vgaarb: loaded [ 0.619268] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.621010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.627311] clocksource: Switched to clocksource kvm-clock [ 0.729860] VFS: Disk quotas dquot_6.6.0 [ 0.731160] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.733887] *** VALIDATE ramfs *** [ 0.735343] *** VALIDATE hugetlbfs *** [ 0.737505] pnp: PnP ACPI init [ 0.739610] pnp: PnP ACPI: found 6 devices [ 0.758115] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.760109] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.762289] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.764689] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.766126] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.767807] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.770226] NET: Registered protocol family 2 [ 0.772573] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.777106] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.780810] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.786300] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.789756] TCP: Hash tables configured (established 65536 bind 65536) [ 0.792670] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.796139] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.799404] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.802437] NET: Registered protocol family 1 [ 0.805454] RPC: Registered named UNIX socket transport module. [ 0.807136] RPC: Registered udp transport module. [ 0.808783] RPC: Registered tcp transport module. [ 0.810359] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.812031] NET: Registered protocol family 44 [ 0.813523] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.815796] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.817863] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.820353] PCI: CLS 0 bytes, default 64 [ 0.822346] Unpacking initramfs... [ 2.115265] debug: unmapping init [mem 0xffff9970bcc54000-0xffff9970bffbffff] [ 2.118801] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.121200] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.124386] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.559059] Initialise system trusted keyrings [ 2.560361] Key type blacklist registered [ 2.561516] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.571096] zbud: loaded [ 2.573955] *** VALIDATE nfs *** [ 2.575156] *** VALIDATE nfs4 *** [ 2.576412] pstore: using deflate compression [ 2.578708] Platform Keyring initialized [ 2.662256] NET: Registered protocol family 38 [ 2.663525] Key type asymmetric registered [ 2.664854] Asymmetric key parser 'x509' registered [ 2.666292] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.669262] io scheduler mq-deadline registered [ 2.670528] io scheduler kyber registered [ 2.671646] io scheduler bfq registered [ 2.672851] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.675109] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.677030] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.678930] ACPI: Power Button [PWRF] [ 2.683405] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.688095] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.697582] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.703456] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.714209] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.739357] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.767349] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.771324] Non-volatile memory driver v1.3 [ 2.772629] Linux agpgart interface v0.103 [ 2.807968] virtio_blk virtio1: [vda] 134584 512-byte logical blocks (68.9 MB/65.7 MiB) [ 2.810537] vda: detected capacity change from 0 to 68907008 [ 2.823676] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.826459] vdb: detected capacity change from 0 to 1073741824 [ 2.838072] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.840919] vdc: detected capacity change from 0 to 2621440000 [ 2.856373] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.858734] vdd: detected capacity change from 0 to 2621440000 [ 2.888655] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.892095] vde: detected capacity change from 0 to 4294967296 [ 2.906516] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.909066] vdf: detected capacity change from 0 to 4294967296 [ 2.943143] libphy: Fixed MDIO Bus: probed [ 2.948608] usbcore: registered new interface driver usbserial_generic [ 2.951575] usbserial: USB Serial support registered for generic [ 2.954094] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.958339] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.960365] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.962871] mousedev: PS/2 mouse device common for all mice [ 2.965903] rtc_cmos 00:05: RTC can wake from S4 [ 2.968861] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.974116] rtc_cmos 00:05: registered as rtc0 [ 2.975369] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.979114] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.983813] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.986391] intel_pstate: CPU model not supported [ 2.992992] hid: raw HID events driver (C) Jiri Kosina [ 2.995305] usbcore: registered new interface driver usbhid [ 2.997368] usbhid: USB HID core driver [ 2.999064] drop_monitor: Initializing network drop monitor service [ 3.001888] Initializing XFRM netlink socket [ 3.003953] NET: Registered protocol family 10 [ 3.007499] Segment Routing with IPv6 [ 3.008558] NET: Registered protocol family 17 [ 3.010324] mpls_gso: MPLS GSO support [ 3.015989] RAS: Correctable Errors collector initialized. [ 3.017687] AVX version of gcm_enc/dec engaged. [ 3.018813] AES CTR mode by8 optimization enabled [ 3.085125] sched_clock: Marking stable (3085078814, 0)->(3837279866, -752201052) [ 3.088541] registered taskstats version 1 [ 3.090192] Loading compiled-in X.509 certificates [ 3.091809] zswap: loaded using pool lzo/zbud [ 3.112083] Key type big_key registered [ 3.121481] Key type encrypted registered [ 3.122583] ima: No TPM chip found, activating TPM-bypass! [ 3.124100] ima: Allocated hash algorithm: sha1 [ 3.125351] ima: No architecture policies found [ 3.126639] evm: Initialising EVM extended attributes: [ 3.127959] evm: security.selinux [ 3.128871] evm: security.ima [ 3.129684] evm: security.capability [ 3.130623] evm: HMAC attrs: 0x1 [ 3.132609] rtc_cmos 00:05: setting system clock to 2026-03-16 13:35:24 UTC (1773668124) [ 3.138365] debug: unmapping init [mem 0xffffffff93a03000-0xffffffff93bfffff] [ 3.140770] debug: unmapping init [mem 0xffffffff92782000-0xffffffff92a58fff] [ 3.149101] Write protecting the kernel read-only data: 28672k [ 3.151873] debug: unmapping init [mem 0xffffffff90e03000-0xffffffff90ffffff] [ 3.154904] debug: unmapping init [mem 0xffffffff91714000-0xffffffff917fffff] [ 3.193567] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.201875] systemd[1]: Detected virtualization kvm. [ 3.203627] systemd[1]: Detected architecture x86-64. [ 3.205205] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.229185] systemd[1]: No hostname configured. [ 3.231053] systemd[1]: Set hostname to . [ 3.232851] random: systemd: uninitialized urandom read (16 bytes read) [ 3.234842] systemd[1]: Initializing machine ID from random generator. [ 3.283634] random: ln: uninitialized urandom read (6 bytes read) [ 3.378932] random: systemd: uninitialized urandom read (16 bytes read) [ 3.381880] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.386786] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.389197] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.854988] device-mapper: uevent: version 1.0.3 [ 3.856914] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 4.192921] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.489294] virtio_net virtio0 ens2: renamed from eth0 [ 4.611033] scsi host0: ata_piix [ 4.657840] scsi host1: ata_piix [ 4.659093] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.661149] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.163831] dracut-initqueue[578]: RTNETLINK answers: File exists [ 9.637414] random: crng init done [ 9.638909] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.885082] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.006626] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.230567] SELinux: Disabled at runtime. [ 11.290293] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.297110] systemd[1]: Detected virtualization kvm. [ 11.298713] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.740852] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.743815] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.748301] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.751041] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.753676] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.761864] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.770480] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ 11.804421] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.175725] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.478530] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.566516] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.612326] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.621347] EDAC sbridge: Ver: 1.1.2 [ 14.404097] Key type dns_resolver registered [ 14.691630] NFS: Registering the id_resolver key type [ 14.693180] Key type id_resolver registered [ 14.695091] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg350-server login: [ 40.470739] libcfs: loading out-of-tree module taints kernel. [ 40.488617] Key type ._llcrypt registered [ 40.489555] Key type .llcrypt registered [ 40.539619] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 47.542523] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 48.092690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 48.098285] alg: No test for adler32 (adler32-zlib) [ 49.068172] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 49.347550] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 50.959259] Key type lgssc registered [ 51.493470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 57.395737] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 70.466341] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 74.734987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 74.743639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 75.841788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 75.857551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 75.896964] Lustre: lustre-MDT0000: new disk, initializing [ 75.932200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 75.942951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 77.328558] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 82.605969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 82.647560] Lustre: 6465:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 82.658868] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 82.661337] Lustre: Skipped 1 previous similar message [ 82.702207] Lustre: lustre-MDT0001: new disk, initializing [ 82.728246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 82.739850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 82.744208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 84.003018] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 86.376117] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 89.547163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 89.642089] Lustre: lustre-OST0000: new disk, initializing [ 89.644040] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 89.662783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 91.432926] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 95.723367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 95.725925] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 95.750992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 96.568726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 96.618920] Lustre: lustre-OST0001: new disk, initializing [ 96.621367] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 96.640178] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 98.658145] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 104.425657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 104.427833] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 104.450804] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 104.484554] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 107.104509] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 113.937524] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing check_logdir /tmp/testlogs/ [ 115.252291] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing yml_node [ 116.740315] Lustre: DEBUG MARKER: Client: 2.17.51.1 [ 117.562706] Lustre: DEBUG MARKER: MDS: 2.17.51.1 [ 118.360113] Lustre: DEBUG MARKER: OSS: 2.17.51.1 [ 118.867949] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Mar 16 09:37:20 EDT 2026 [ 124.466805] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 124.916139] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 139.231736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 139.236362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 139.241168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 140.255652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 140.256026] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 140.258377] Lustre: Skipped 2 previous similar messages [ 140.262791] Lustre: Skipped 2 previous similar messages [ 142.628673] Lustre: server umount lustre-MDT0000 complete [ 144.003750] LustreError: 7396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668265 with bad export cookie 5667010490263005279 [ 144.005437] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 144.007941] LustreError: 7396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 145.376582] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 145.378116] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 145.381597] Lustre: Skipped 1 previous similar message [ 145.384255] Lustre: Skipped 1 previous similar message [ 149.407519] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 149.409181] Lustre: Skipped 1 previous similar message [ 150.187742] Lustre: server umount lustre-MDT0001 complete [ 157.695207] Lustre: server umount lustre-OST0000 complete [ 165.065230] Lustre: server umount lustre-OST0001 complete [ 167.052681] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 170.122780] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 182.336227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.438788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.453237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.503991] Lustre: lustre-MDT0000: new disk, initializing [ 182.535443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.546636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 183.917382] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 188.463473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.513220] Lustre: 14927:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 188.636873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 190.050705] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 193.396288] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 194.582655] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 196.952243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.072296] Lustre: lustre-OST0000: new disk, initializing [ 197.073600] Lustre: Skipped 1 previous similar message [ 197.075726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 197.078888] Lustre: Skipped 2 previous similar messages [ 197.121843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.125454] Lustre: Skipped 1 previous similar message [ 198.272851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 198.275762] Lustre: Skipped 1 previous similar message [ 198.277670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 198.320299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.241975] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 202.580618] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 204.036089] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 204.112574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 205.398567] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 205.469766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 208.865136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 208.865417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 208.867456] LustreError: Skipped 1 previous similar message [ 208.871365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 208.873687] Lustre: Skipped 1 previous similar message [ 212.244459] Lustre: server umount lustre-OST0000 complete [ 213.984425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 213.989452] Lustre: Skipped 2 previous similar messages [ 219.104108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 219.107212] Lustre: Skipped 3 previous similar messages [ 219.613788] Lustre: server umount lustre-MDT0000 complete [ 220.950408] LustreError: 16670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668342 with bad export cookie 5667010490263006798 [ 220.953102] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 220.953543] LustreError: 16670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 221.076185] Lustre: server umount lustre-MDT0001 complete [ 222.513040] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:39:04 (1773668344) [ 223.410381] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 224.033521] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:39:05 (1773668345) [ 226.342077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.524583] LustreError: 18927:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 226.533281] LustreError: 18927:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 226.555289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 227.893556] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 230.929494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 232.485322] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 234.436699] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 235.710412] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 238.274598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.428643] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 242.403294] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 256.996710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 257.003837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 257.010874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 261.090652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 262.548537] Lustre: server umount lustre-OST0000 complete [ 270.235038] Lustre: server umount lustre-MDT0000 complete [ 271.328355] LustreError: 19640:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 271.333243] LustreError: 19640:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 271.603419] LustreError: 18910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668392 with bad export cookie 5667010490263007680 [ 271.606411] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 271.607501] LustreError: 18910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 271.723738] Lustre: server umount lustre-MDT0001 complete [ 275.461302] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 276.618468] Key type lgssc unregistered [ 276.764379] LNet: 22125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 276.768025] LNetError: 22125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 276.776302] LNet: Removed LNI 192.168.203.150@tcp [ 277.125137] Key type .llcrypt unregistered [ 277.126822] Key type ._llcrypt unregistered [ 285.869389] Key type ._llcrypt registered [ 285.870468] Key type .llcrypt registered [ 285.917949] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 286.246614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 286.278096] alg: No test for adler32 (adler32-zlib) [ 287.136383] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 287.221542] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 288.807152] Key type lgssc registered [ 289.187713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 292.351475] loop: module loaded [ 292.376538] loop0: detected capacity change from 0 to 204800000 [ 292.496035] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 292.501108] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 292.629743] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 294.914387] loop0: detected capacity change from 0 to 204800000 [ 294.983271] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 294.987896] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 294.991513] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 296.897585] loop0: detected capacity change from 0 to 204800000 [ 298.874590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 298.881475] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 299.983314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 299.999838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 300.038906] Lustre: lustre-MDT0000: new disk, initializing [ 300.063633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 300.072273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 301.554504] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 305.285964] loop1: detected capacity change from 0 to 204800000 [ 307.376126] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 307.488785] Lustre: lustre-OST07c6: new disk, initializing [ 307.491424] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 307.493405] Lustre: Skipped 1 previous similar message [ 307.516416] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 309.490595] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 313.347836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 313.352512] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 313.363729] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 317.019588] Lustre: 24258:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 317.022931] Lustre: 24258:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 317.025359] Lustre: 24258:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 317.028442] Lustre: 24258:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 317.031784] Lustre: 24258:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 317.034554] Lustre: 24258:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 332.292087] Lustre: 24259:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 332.295197] Lustre: 24259:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 332.297379] Lustre: 24259:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 332.299873] Lustre: 24259:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 332.303418] Lustre: 24259:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 332.305833] Lustre: 24259:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 335.955116] Lustre: 24287:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 335.959655] Lustre: 24287:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 335.962450] Lustre: 24287:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 335.965557] Lustre: 24287:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 335.968113] Lustre: 24287:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 335.970834] Lustre: 24287:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 335.973413] Lustre: 24287:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 335.976247] Lustre: 24287:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 335.979111] Lustre: 24287:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 335.982754] Lustre: 24287:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 335.985642] Lustre: 24287:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 335.988491] Lustre: 24287:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 339.050250] Lustre: 24257:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 339.054603] Lustre: 24257:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 339.057399] Lustre: 24257:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 339.060274] Lustre: 24257:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 339.062266] Lustre: 24257:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 339.065084] Lustre: 24257:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 339.067982] Lustre: 24257:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 339.071275] Lustre: 24257:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 339.073483] Lustre: 24257:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 339.075871] Lustre: 24257:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 339.079030] Lustre: 24257:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 339.081501] Lustre: 24257:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 345.670101] Lustre: 24259:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 345.673428] Lustre: 24259:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 3 previous similar messages [ 345.675727] Lustre: 24259:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 345.678737] Lustre: 24259:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 345.681527] Lustre: 24259:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 345.685100] Lustre: 24259:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 345.687751] Lustre: 24259:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 345.690586] Lustre: 24259:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 345.693930] Lustre: 24259:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 345.698086] Lustre: 24259:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 345.701549] Lustre: 24259:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 345.704991] Lustre: 24259:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 355.346742] Lustre: 24257:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 355.350877] Lustre: 24257:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 355.352955] Lustre: 24257:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 355.356758] Lustre: 24257:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 355.360708] Lustre: 24257:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 355.364570] Lustre: 24257:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 355.368411] Lustre: 24257:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 355.370820] Lustre: 24257:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 355.373203] Lustre: 24257:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 355.375527] Lustre: 24257:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 355.378120] Lustre: 24257:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 355.380838] Lustre: 24257:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 400.352547] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 400.352602] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 400.361748] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 402.391317] Lustre: server umount lustre-OST07c6 complete [ 406.142184] Lustre: server umount lustre-MDT0000 complete [ 412.207252] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 413.364785] Key type lgssc unregistered [ 413.509882] LNet: 27777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.514466] LNetError: 27777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 413.525318] LNet: Removed LNI 192.168.203.150@tcp [ 413.861523] Key type .llcrypt unregistered [ 413.862694] Key type ._llcrypt unregistered [ 424.018527] Key type ._llcrypt registered [ 424.020795] Key type .llcrypt registered [ 424.123510] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 424.745288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 424.783309] alg: No test for adler32 (adler32-zlib) [ 425.829831] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 426.162277] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 427.832564] Key type lgssc registered [ 429.186883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 436.522373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 436.535481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.847777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.788682] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 447.970833] LustreError: 29010:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 447.984386] LustreError: 29010:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 448.442740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.705747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 451.975491] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 456.775662] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 459.702806] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 465.431096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.755458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.971383] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 476.769899] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 483.200195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 508.896466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 508.902675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.905512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 508.941689] Lustre: Skipped 1 previous similar message [ 514.023442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 514.047659] Lustre: Skipped 2 previous similar messages [ 519.136713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 519.146093] Lustre: Skipped 1 previous similar message [ 521.183153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 521.306577] Lustre: server umount lustre-OST0000 complete [ 524.256186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.266818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 524.270710] Lustre: Skipped 1 previous similar message [ 529.378543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 529.381251] Lustre: Skipped 1 previous similar message [ 530.461149] Lustre: server umount lustre-MDT0000 complete [ 533.281623] LustreError: 28991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668654 with bad export cookie 4541625746883248614 [ 533.283187] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.290257] LustreError: 28991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 533.551435] Lustre: server umount lustre-MDT0001 complete [ 540.533733] hrtimer: interrupt took 4810173 ns [ 541.751601] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 543.597041] Key type lgssc unregistered [ 543.797817] LNet: 32414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 543.803219] LNetError: 32414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 543.817421] LNet: Removed LNI 192.168.203.150@tcp [ 544.369893] Key type .llcrypt unregistered [ 544.371877] Key type ._llcrypt unregistered [ 556.983607] Key type ._llcrypt registered [ 556.985933] Key type .llcrypt registered [ 557.098579] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 557.842306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 557.872848] alg: No test for adler32 (adler32-zlib) [ 558.868456] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 559.070433] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 560.712337] Key type lgssc registered [ 561.597641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 567.651368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 567.668712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.042265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.853948] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 579.521801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.557992] LustreError: 33644:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.577340] LustreError: 33644:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 579.846269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 583.130713] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 588.093714] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 591.031802] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 596.623287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 596.875329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.591506] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 605.441281] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 614.325455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 624.619367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 624.627973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 624.628129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 624.643556] Lustre: Skipped 1 previous similar message [ 629.466772] Lustre: server umount lustre-OST0000 complete [ 634.853962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.865190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 634.865469] Lustre: Skipped 2 previous similar messages [ 638.133398] Lustre: server umount lustre-MDT0000 complete [ 639.972910] LustreError: 34850:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.985640] LustreError: 34850:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 640.367246] LustreError: 33626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668761 with bad export cookie 16626597774525217452 [ 640.374726] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.578687] Lustre: server umount lustre-MDT0001 complete [ 646.059775] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 647.698943] Key type lgssc unregistered [ 647.889322] LNet: 36841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.895196] LNetError: 36841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.912429] LNet: Removed LNI 192.168.203.150@tcp [ 648.473751] Key type .llcrypt unregistered [ 648.475137] Key type ._llcrypt unregistered [ 665.879449] Key type ._llcrypt registered [ 665.882334] Key type .llcrypt registered [ 665.956505] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 666.887235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 666.908297] alg: No test for adler32 (adler32-zlib) [ 667.913437] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 668.075484] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 669.703158] Key type lgssc registered [ 670.615822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 676.331611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 676.346925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 677.665184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 681.137254] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 686.601747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.800328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 688.951135] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 691.645175] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 693.505850] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 696.760811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 696.928255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 700.070835] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 702.842809] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 703.087327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 707.552519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 707.561414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.574979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.581891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.585686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.593514] Lustre: Skipped 1 previous similar message [ 709.698696] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 709.714263] Lustre: Skipped 2 previous similar messages [ 713.087828] Lustre: server umount lustre-MDT0000 complete [ 713.697172] LustreError: 38218:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.708589] LustreError: 38218:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 714.784064] LustreError: 38213:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 715.410281] LustreError: 38201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668836 with bad export cookie 12044496987542028660 [ 715.413695] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 715.416973] LustreError: 38201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 718.816980] LustreError: 39833:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 718.819475] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.835333] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 720.867674] LustreError: 38214:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.881564] LustreError: 38214:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 721.698068] Lustre: server umount lustre-MDT0001 complete [ 741.478303] Lustre: server umount lustre-OST0000 complete [ 752.517736] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 754.368469] Key type lgssc unregistered [ 754.579307] LNet: 41722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.585389] LNetError: 41722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 754.597603] LNet: Removed LNI 192.168.203.150@tcp [ 755.129898] Key type .llcrypt unregistered [ 755.131646] Key type ._llcrypt unregistered [ 769.902523] Key type ._llcrypt registered [ 769.907200] Key type .llcrypt registered [ 770.004244] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 770.496890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 770.570789] alg: No test for adler32 (adler32-zlib) [ 771.506082] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 771.622518] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 773.239255] Key type lgssc registered [ 774.011772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 779.314746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 779.330356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 780.665446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 783.162678] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 788.685337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 788.943918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 791.938976] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 795.827256] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 798.201294] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 802.691481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 802.850928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.903777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 805.529670] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 808.394015] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 819.186207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 819.189485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.200545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 819.207378] Lustre: Skipped 1 previous similar message [ 820.273877] Lustre: lustre-OST0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 822.830837] Lustre: server umount lustre-OST0000 complete [ 838.811916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.816990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 838.823098] Lustre: Skipped 2 previous similar messages [ 845.083479] Lustre: server umount lustre-MDT0000 complete [ 845.280102] LustreError: 43101:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 845.287287] LustreError: 43101:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 847.403453] LustreError: 43082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668968 with bad export cookie 17126932350905283773 [ 847.406359] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 847.409642] LustreError: 43082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 853.726140] Lustre: server umount lustre-MDT0001 complete [ 860.550340] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 862.522487] Key type lgssc unregistered [ 862.691979] LNet: 46452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.699388] LNetError: 46452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.715820] LNet: Removed LNI 192.168.203.150@tcp [ 863.242378] Key type .llcrypt unregistered [ 863.244497] Key type ._llcrypt unregistered [ 877.115804] Key type ._llcrypt registered [ 877.117293] Key type .llcrypt registered [ 877.185657] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 877.922140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 877.939163] alg: No test for adler32 (adler32-zlib) [ 878.887565] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 879.021630] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 880.623176] Key type lgssc registered [ 881.190832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 886.192982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 886.208084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.521727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.265833] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 895.417575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.615261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 898.028935] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 901.146545] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 902.936171] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 906.562663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 906.762781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 909.319669] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 911.934314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 911.948847] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 925.728434] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 927.749934] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 938.853396] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-16 9h50m59s [ 939.657467] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 941.535962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 941.538984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.544845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 942.562147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 942.567772] Lustre: Skipped 4 previous similar messages [ 945.200072] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 946.839101] Lustre: server umount lustre-MDT0000 complete [ 948.707779] LustreError: 47831:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.725525] LustreError: 47831:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 948.833682] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 951.324531] LustreError: 47825:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.418512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.485581] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 952.617588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 952.657541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 953.639627] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 954.642651] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 956.618860] Lustre: DEBUG MARKER: Wait for df (35965) ... [ 957.441611] Lustre: DEBUG MARKER: done [ 960.992503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 961.000060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 961.012138] Lustre: Skipped 2 previous similar messages [ 961.015766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 961.019862] Lustre: Skipped 1 previous similar message [ 963.044428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 963.051700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 965.521915] Lustre: server umount lustre-OST0000 complete [ 969.184909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.186719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 969.190742] Lustre: Skipped 1 previous similar message [ 969.191076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 969.194630] Lustre: Skipped 4 previous similar messages [ 974.015821] Lustre: server umount lustre-MDT0000 complete [ 975.265539] LustreError: 47831:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.275467] LustreError: 47831:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 976.179976] LustreError: 47810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669097 with bad export cookie 13791196836887155318 [ 976.188417] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 976.372371] Lustre: server umount lustre-MDT0001 complete [ 981.409066] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 982.931428] Key type lgssc unregistered [ 983.087569] LNet: 52406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.091559] LNetError: 52406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.100613] LNet: Removed LNI 192.168.203.150@tcp [ 983.551896] Key type .llcrypt unregistered [ 983.553726] Key type ._llcrypt unregistered [ 1007.503555] Key type ._llcrypt registered [ 1007.505309] Key type .llcrypt registered [ 1007.555230] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1008.076887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1008.087064] alg: No test for adler32 (adler32-zlib) [ 1008.991254] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 1009.137653] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1010.751176] Key type lgssc registered [ 1011.338860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1015.687479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1015.703633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.923100] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1016.947752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1017.043559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1019.413349] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1024.674043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.746277] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1024.765458] Lustre: 54574:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1024.930915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1027.475574] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1031.457692] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1033.922546] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1038.068937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1038.171419] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1038.327593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.879689] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1045.081900] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1045.573693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1050.585597] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-16 9h52m51s [ 1051.612366] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1056.651761] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1059.295629] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1059.357531] LustreError: 56662:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1079.922531] Lustre: lustre-MDT0000: Client 956f1538-e53b-4947-b4f3-70c38cca42c3 (at 192.168.203.50@tcp) reconnecting [ 1099.359159] LustreError: 56662:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1102.522492] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1105.938510] Lustre: DEBUG MARKER: done [ 1117.156176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.159510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.160570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1117.172200] Lustre: Skipped 1 previous similar message [ 1117.193197] Lustre: Skipped 1 previous similar message [ 1119.867474] Lustre: server umount lustre-OST0000 complete [ 1127.399975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.403523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1127.407100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1127.410751] Lustre: Skipped 1 previous similar message [ 1127.450572] LustreError: Skipped 1 previous similar message [ 1128.487227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1128.506305] Lustre: Skipped 2 previous similar messages [ 1134.561366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1134.565892] Lustre: Skipped 1 previous similar message [ 1138.146921] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1138.245544] Lustre: server umount lustre-MDT0000 complete [ 1140.776921] LustreError: 54566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669262 with bad export cookie 196323590330774267 [ 1140.785313] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1140.790130] LustreError: 54566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1141.105791] Lustre: server umount lustre-MDT0001 complete [ 1150.015267] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1152.370792] Key type lgssc unregistered [ 1152.674741] LNet: 58541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.687870] LNetError: 58541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1153.766404] LNet: Removed LNI 192.168.203.150@tcp [ 1154.530416] Key type .llcrypt unregistered [ 1154.533114] Key type ._llcrypt unregistered [ 1190.600333] Key type ._llcrypt registered [ 1190.603207] Key type .llcrypt registered [ 1190.754717] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1192.346733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1192.395067] alg: No test for adler32 (adler32-zlib) [ 1193.519406] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 1193.782730] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1195.463171] Key type lgssc registered [ 1196.546879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1199.985975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1200.007668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1201.392517] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1201.444563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1201.670702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1202.878787] Lustre: Failing over lustre-MDT0000 [ 1203.136906] Lustre: server umount lustre-MDT0000 complete [ 1209.605843] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:55:30 (1773669330) [ 1215.601315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.976259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.960450] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1226.596804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.654763] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1226.670971] Lustre: 61306:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1226.920067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1230.153716] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1234.293225] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1237.064547] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1241.088633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1241.202197] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1241.338443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1245.070594] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1248.840806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1248.874585] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1253.047568] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1258.982661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.986636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.991703] Lustre: Skipped 1 previous similar message [ 1259.002543] Lustre: Skipped 2 previous similar messages [ 1260.392508] Lustre: server umount lustre-MDT0000 complete [ 1263.643469] LustreError: 61298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669385 with bad export cookie 9583859734490289273 [ 1263.658910] LustreError: 61298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1263.660745] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1264.075925] Lustre: server umount lustre-MDT0001 complete [ 1266.729344] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1270.453477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1277.342213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.479367] Lustre: 60229:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773669385/real 1773669385] req@000000006b3bcb85 x1859827075678976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773669401 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1280.496188] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1280.502182] Lustre: Skipped 1 previous similar message [ 1284.575513] Lustre: 60230:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773669389/real 1773669389] req@000000009bc57f90 x1859827075679104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773669405 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1285.600277] Lustre: 60228:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773669390/real 1773669390] req@000000007779c083 x1859827075679232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773669406 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1289.049111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1289.117801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1290.015226] Lustre: 60227:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773669395/real 1773669395] req@00000000550f7ab6 x1859827075679488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773669411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1291.941976] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1295.199530] Lustre: 60230:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773669400/real 1773669400] req@0000000060026bdd x1859827075679616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773669416 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1295.330637] LustreError: 64736:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.339356] LustreError: 64736:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 1297.567775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1297.908813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1298.927815] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1300.387360] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1303.734762] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1306.006834] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1312.923552] Lustre: server umount lustre-MDT0000 complete [ 1313.248531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1313.251329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.258605] LustreError: 62926:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1314.279435] LustreError: 62927:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1314.287472] LustreError: 62927:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1315.819237] LustreError: 62932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669437 with bad export cookie 9583859734490291548 [ 1315.820687] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1315.830399] LustreError: 62932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1316.084401] Lustre: server umount lustre-MDT0001 complete [ 1321.569598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1328.873908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.687147] Lustre: 60229:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773669440/real 1773669440] req@000000008aa8a817 x1859827075702912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773669456 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1334.713122] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.729094] Lustre: Skipped 2 previous similar messages [ 1340.899874] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8500b5a6ab76a2cd [ 1340.903445] Lustre: MGC192.168.203.150@tcp: Connection restored to 0@lo (at 0@lo) [ 1340.906047] Lustre: Skipped 1 previous similar message [ 1341.178665] LustreError: 62927:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.248655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.288057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1344.979514] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1345.317881] LustreError: 67193:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.327445] LustreError: 67193:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1351.580254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.037648] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1355.296187] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1359.177330] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1361.512026] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1367.522027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1367.529476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.544246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1369.571187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1371.810188] Lustre: server umount lustre-MDT0000 complete [ 1374.076116] LustreError: 61297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669495 with bad export cookie 9583859734490292941 [ 1374.082244] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1374.088340] LustreError: 61297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1374.294057] Lustre: server umount lustre-MDT0001 complete [ 1377.995643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1380.233852] Lustre: DEBUG MARKER: files compared the same [ 1390.901643] Lustre: server umount lustre-OST0000 complete [ 1401.229697] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1402.768815] Key type lgssc unregistered [ 1402.982546] LNet: 70224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1402.991278] LNetError: 70224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1403.013636] LNet: Removed LNI 192.168.203.150@tcp [ 1403.480978] Key type .llcrypt unregistered [ 1403.482534] Key type ._llcrypt unregistered [ 1417.418442] Key type ._llcrypt registered [ 1417.419710] Key type .llcrypt registered [ 1417.485925] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1417.961191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1417.976183] alg: No test for adler32 (adler32-zlib) [ 1418.877129] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 1419.011953] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1420.631153] Key type lgssc registered [ 1421.343283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1425.731496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1425.740825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1426.986203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1428.864715] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug +malloc all [ 1433.594462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1433.820302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1435.840265] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug +malloc all [ 1439.038368] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1440.931523] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1444.554872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1444.796300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.233493] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug +malloc all [ 1451.089231] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1451.940307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1467.359830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1467.362750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.367497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1469.408151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1469.409219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1469.409385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.427731] Lustre: Skipped 1 previous similar message [ 1471.788220] Lustre: server umount lustre-OST0000 complete [ 1474.530271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.538620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1479.647599] LustreError: 71602:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.660507] LustreError: 71602:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1479.674698] Lustre: server umount lustre-MDT0000 complete [ 1481.576767] LustreError: 71582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669602 with bad export cookie 10503573237751744780 [ 1481.578344] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1481.580965] LustreError: 71582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1481.771246] Lustre: server umount lustre-MDT0001 complete [ 1487.303287] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1488.803789] Key type lgssc unregistered [ 1488.976424] LNet: 74798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.982515] LNetError: 74798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1488.992625] LNet: Removed LNI 192.168.203.150@tcp [ 1489.518580] Key type .llcrypt unregistered [ 1489.520379] Key type ._llcrypt unregistered [ 1502.658661] Key type ._llcrypt registered [ 1502.660895] Key type .llcrypt registered [ 1502.727952] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1503.328146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1503.345631] alg: No test for adler32 (adler32-zlib) [ 1504.289408] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 1504.436388] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1506.031160] Key type lgssc registered [ 1506.575755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1510.692415] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1510.704864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1526.175523] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1531.298313] LustreError: 76151:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 1557.919172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1559.754356] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1561.438590] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1563.756969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1563.797272] Lustre: *** cfs_fail_loc=706, val=0*** [ 1563.944651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1563.970733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1565.157555] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1567.944953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.065497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1569.406809] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1571.254211] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1572.466131] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1573.344860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1573.346951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1573.354530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1573.356390] Lustre: Skipped 1 previous similar message [ 1578.465646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1578.468553] Lustre: Skipped 1 previous similar message [ 1579.274812] Lustre: server umount lustre-OST0000 complete [ 1583.585891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1583.586239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1583.589408] Lustre: Skipped 1 previous similar message [ 1583.589911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1583.594607] Lustre: Skipped 1 previous similar message [ 1586.586858] Lustre: server umount lustre-MDT0000 complete [ 1587.777445] LustreError: 78486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669709 with bad export cookie 14665515093800816714 [ 1587.781136] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1587.883489] Lustre: server umount lustre-MDT0001 complete [ 1590.948723] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1591.900596] Key type lgssc unregistered [ 1592.022426] LNet: 79425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1592.024893] LNetError: 79425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1592.033272] LNet: Removed LNI 192.168.203.150@tcp [ 1592.314211] Key type .llcrypt unregistered [ 1592.315366] Key type ._llcrypt unregistered [ 1601.058791] Key type ._llcrypt registered [ 1601.060354] Key type .llcrypt registered [ 1601.099505] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1601.500262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1601.509342] alg: No test for adler32 (adler32-zlib) [ 1602.376513] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 1602.468682] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1604.047171] Key type lgssc registered [ 1604.423254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1606.814676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1606.819945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1609.032912] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1611.319261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.472692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1612.622783] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1615.240815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1615.367527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.117731] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1619.680868] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1619.686175] Lustre: MGS: Client c9310d1c-a0a2-4814-92a6-b58fca56cbb3 (at 0@lo) reconnecting [ 1619.763629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1619.797379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1620.956271] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1630.175940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1630.178028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.181510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1631.202391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1631.202441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.207555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1634.702437] Lustre: server umount lustre-OST0000 complete [ 1636.320504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1636.320724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.326116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1636.327985] Lustre: Skipped 2 previous similar messages [ 1641.440040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1641.442844] Lustre: Skipped 1 previous similar message [ 1646.559930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1646.562756] Lustre: Skipped 1 previous similar message [ 1650.143094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1650.185794] Lustre: server umount lustre-MDT0000 complete [ 1650.657689] LustreError: 82979:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.665991] LustreError: 82979:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 1651.525638] Lustre: server umount lustre-MDT0001 complete [ 1652.854771] Lustre: server umount MGS complete [ 1656.614887] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:02:58 (1773669778) [ 1666.295471] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 1669.202491] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1679.740416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1680.903910] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1684.427349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.504738] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1684.518956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1684.550859] Lustre: lustre-MDT0001: new disk, initializing [ 1684.572555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1684.581051] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1684.583727] Lustre: 88530:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1684.586442] LustreError: 88530:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1685.728885] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1689.609085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1689.704448] Lustre: lustre-OST0000: new disk, initializing [ 1689.706992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1689.730332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1691.456398] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1694.296582] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1696.287725] Lustre: 89999:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000001b4965a8 x1859827503498880/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773669833 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1696.356939] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1696.360635] Lustre: MGS: Client 187c18dc-5c76-4d33-80ab-6a5d6b4200b2 (at 0@lo) reconnecting [ 1696.374176] Lustre: 90209:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1696.380302] LustreError: 88516:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1696.388890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1696.420956] Lustre: lustre-MDT0000: new disk, initializing [ 1696.468083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1697.503624] Lustre: 90236:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000a39ebdc3 x1859827503504128/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1773669834 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1697.572363] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1698.335469] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1702.562100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1702.565350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1702.632270] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1722.338649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.340263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1722.342970] Lustre: Skipped 2 previous similar messages [ 1727.632693] Lustre: server umount lustre-OST0000 complete [ 1732.578127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1732.578770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.585218] Lustre: Skipped 1 previous similar message [ 1742.816061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1742.818771] Lustre: Skipped 8 previous similar messages [ 1743.327138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1743.365350] Lustre: server umount lustre-MDT0000 complete [ 1750.764300] Lustre: server umount lustre-MDT0001 complete [ 1752.003687] Lustre: server umount MGS complete [ 1755.383957] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:04:36 (1773669876) [ 1760.618938] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1761.571989] Key type lgssc unregistered [ 1761.699357] LNet: 92695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.701552] LNetError: 92695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.712359] LNet: Removed LNI 192.168.203.150@tcp [ 1762.006653] Key type .llcrypt unregistered [ 1762.007664] Key type ._llcrypt unregistered [ 1768.534522] Key type ._llcrypt registered [ 1768.535547] Key type .llcrypt registered [ 1768.573654] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1768.984437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1769.030228] alg: No test for adler32 (adler32-zlib) [ 1769.872784] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 1769.950819] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1771.527203] Key type lgssc registered [ 1771.917664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1774.614035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1774.622977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1775.794663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.960809] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1779.636792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.741510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1780.899662] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1782.552434] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1783.624343] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1785.645888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1785.754415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.790416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1787.473937] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1789.120394] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1795.044087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.045619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1795.045892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1795.047455] Lustre: Skipped 1 previous similar message [ 1798.606765] Lustre: server umount lustre-OST0000 complete [ 1800.160323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1800.160657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.163037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.168509] Lustre: Skipped 1 previous similar message [ 1800.169892] Lustre: Skipped 1 previous similar message [ 1805.280155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1805.282068] Lustre: Skipped 2 previous similar messages [ 1805.969883] Lustre: server umount lustre-MDT0000 complete [ 1807.278375] LustreError: 96321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669928 with bad export cookie 14926553168798262504 [ 1807.281129] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1807.281719] LustreError: 96321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1807.410441] Lustre: server umount lustre-MDT0001 complete [ 1810.826761] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1811.861921] Key type lgssc unregistered [ 1811.977366] LNet: 97054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.980824] LNetError: 97054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1811.988320] LNet: Removed LNI 192.168.203.150@tcp [ 1812.276814] Key type .llcrypt unregistered [ 1812.277766] Key type ._llcrypt unregistered [ 1818.933711] Key type ._llcrypt registered [ 1818.934763] Key type .llcrypt registered [ 1818.976316] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1819.380772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1819.445143] alg: No test for adler32 (adler32-zlib) [ 1820.310797] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 1820.400526] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1821.983135] Key type lgssc registered [ 1822.341792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1824.410964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1824.412222] LustreError: 98098:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1824.413282] Lustre: Skipped 1 previous similar message [ 1824.417386] LustreError: 98097:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1824.419751] LustreError: 98098:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1824.421763] LustreError: 98098:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1824.425189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1824.425469] LustreError: 98098:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1825.587306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1828.064549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.159807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1829.431240] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1831.102608] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing lsmod [ 1832.143770] LustreError: 99267:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1832.144093] LustreError: 99268:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1832.150210] LustreError: 99267:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1832.153362] LustreError: 99267:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1832.156912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1832.158222] LustreError: 99267:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1832.309481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.298921] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.299478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.306117] Lustre: Skipped 1 previous similar message [ 1838.432969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1838.434699] Lustre: Skipped 4 previous similar messages [ 1839.450556] Lustre: server umount lustre-MDT0000 complete [ 1840.732352] LustreError: 98111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669962 with bad export cookie 3129489878519444954 [ 1840.735779] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1840.736257] LustreError: 98111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1842.527263] LustreError: 98565:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b9a61dc4 x1859827732069120/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1842.582724] Lustre: server umount lustre-MDT0001 complete [ 1854.108932] Lustre: server umount lustre-OST0000 complete [ 1857.109934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1857.297083] LustreError: 100335:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1857.303287] LustreError: 100335:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1857.324407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.779993] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1861.505400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.623675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1862.795666] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1864.519034] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1865.659146] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1867.758155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1869.563235] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1871.168633] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1882.212641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1891.808245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1891.810384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.814826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1892.833499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1892.833662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1894.027982] Lustre: server umount lustre-OST0000 complete [ 1897.439806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1897.442449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.446591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1897.450337] Lustre: Skipped 2 previous similar messages [ 1901.398046] Lustre: server umount lustre-MDT0000 complete [ 1902.697099] LustreError: 100315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670024 with bad export cookie 3129489878519445612 [ 1902.699629] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1902.702593] LustreError: 100315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1902.816828] Lustre: server umount lustre-MDT0001 complete [ 1906.667907] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1907.697561] Key type lgssc unregistered [ 1907.828334] LNet: 103542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1907.830809] LNetError: 103542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1907.837258] LNet: Removed LNI 192.168.203.150@tcp [ 1908.129037] Key type .llcrypt unregistered [ 1908.130067] Key type ._llcrypt unregistered [ 1916.292754] Key type ._llcrypt registered [ 1916.293758] Key type .llcrypt registered [ 1916.327059] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1916.614751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1916.641285] alg: No test for adler32 (adler32-zlib) [ 1917.477222] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 1917.550471] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1919.119161] Key type lgssc registered [ 1919.454971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1921.876553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1921.882394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.024591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.169893] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1926.804531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1926.917109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1928.090877] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1929.741769] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1930.846652] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1932.878366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1932.978635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.990212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1934.663988] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1936.291771] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1939.139595] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1939.876323] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1942.497083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1942.498985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1942.502562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1942.503842] Lustre: Skipped 1 previous similar message [ 1946.893990] Lustre: server umount lustre-OST0000 complete [ 1952.736683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1952.736834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1952.737246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1952.739671] Lustre: Skipped 2 previous similar messages [ 1954.195056] Lustre: server umount lustre-MDT0000 complete [ 1955.279431] LustreError: 104897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670076 with bad export cookie 16008453641164955813 [ 1955.282450] LustreError: 104897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1955.283302] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1955.377919] Lustre: server umount lustre-MDT0001 complete [ 1958.530659] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1959.446125] Key type lgssc unregistered [ 1959.562337] LNet: 108196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.564258] LNetError: 108196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.573248] LNet: Removed LNI 192.168.203.150@tcp [ 1959.815219] Key type .llcrypt unregistered [ 1959.816302] Key type ._llcrypt unregistered [ 1966.059850] Key type ._llcrypt registered [ 1966.060914] Key type .llcrypt registered [ 1966.094693] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1966.355147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1966.364237] alg: No test for adler32 (adler32-zlib) [ 1967.211915] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 1967.291199] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1968.871126] Key type lgssc registered [ 1969.185858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1971.410750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1971.416527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.586829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1973.769769] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1976.357848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.466695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1977.830662] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1979.524307] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1980.702626] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1982.906869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1982.945629] LustreError: 110969:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.952195] LustreError: 110969:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1983.012357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.664457] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1986.086529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1986.341754] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1991.139689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1991.140210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1991.143923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1991.145107] LustreError: Skipped 1 previous similar message [ 1991.147025] Lustre: Skipped 1 previous similar message [ 1995.980829] Lustre: server umount lustre-OST0000 complete [ 1997.280074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1997.283947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.288036] Lustre: Skipped 1 previous similar message [ 1997.290655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2001.376299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2001.376832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.350862] Lustre: server umount lustre-MDT0000 complete [ 2004.550116] LustreError: 109360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670125 with bad export cookie 13756902270776360600 [ 2004.552826] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.553567] LustreError: 109360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2004.667971] Lustre: server umount lustre-MDT0001 complete [ 2008.306304] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2009.248650] Key type lgssc unregistered [ 2009.374291] LNet: 112555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.377255] LNetError: 112555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.383299] LNet: Removed LNI 192.168.203.150@tcp [ 2009.645535] Key type .llcrypt unregistered [ 2009.646525] Key type ._llcrypt unregistered [ 2018.277252] Key type ._llcrypt registered [ 2018.278420] Key type .llcrypt registered [ 2018.313186] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2018.742154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2018.760900] alg: No test for adler32 (adler32-zlib) [ 2019.607369] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 2019.692942] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2021.271102] Key type lgssc registered [ 2021.632152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2024.167796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2024.176546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.362704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.798308] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2029.543317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2029.649329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2030.834992] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2032.414923] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2033.631212] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2035.685056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2035.779509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2037.510211] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2039.112672] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2051.180633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2057.139654] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2058.443204] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2058.446055] Lustre: Skipped 2 previous similar messages [ 2059.753487] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2059.757549] Lustre: Skipped 2 previous similar messages [ 2066.430893] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2066.442780] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2068.567086] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.50@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2068.569934] Lustre: Skipped 2 previous similar messages [ 2073.590981] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.50@tcp 192.168.0.[2,10]@tcp [ 2075.244912] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.50/24@tcp in log lustre-MDT0000 [ 2075.249433] Lustre: Skipped 2 previous similar messages [ 2080.246671] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.50/24@tcp [ 2080.249103] Lustre: Skipped 1 previous similar message [ 2085.343962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2085.347481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2085.351630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2087.394632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2087.394905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2087.395056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2087.404854] Lustre: Skipped 1 previous similar message [ 2088.976286] Lustre: server umount lustre-OST0000 complete [ 2090.975971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2090.979012] LustreError: Skipped 1 previous similar message [ 2090.980719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.986078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2096.406617] Lustre: server umount lustre-MDT0000 complete [ 2097.632332] LustreError: 113932:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2097.637193] LustreError: 113932:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2097.670667] LustreError: 116245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670219 with bad export cookie 14107813682543756778 [ 2097.674121] LustreError: 116245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2097.675644] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2097.794676] Lustre: server umount lustre-MDT0001 complete [ 2101.360519] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2102.410633] Key type lgssc unregistered [ 2102.552369] LNet: 118775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2102.554647] LNetError: 118775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.564447] LNet: Removed LNI 192.168.203.150@tcp [ 2102.873777] Key type .llcrypt unregistered [ 2102.874963] Key type ._llcrypt unregistered [ 2113.025261] Key type ._llcrypt registered [ 2113.026304] Key type .llcrypt registered [ 2113.073496] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2113.494921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2113.562911] alg: No test for adler32 (adler32-zlib) [ 2114.408828] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 2114.491743] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2116.071136] Key type lgssc registered [ 2116.419816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2118.923269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2118.930186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.076129] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2120.082594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2120.084491] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.50/24@tcp [ 2121.362728] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2124.170406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2124.272934] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2124.280954] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.50/24@tcp [ 2124.295286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2125.579902] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2127.332374] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2128.476467] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2130.598915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2130.711449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.568105] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2134.265904] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2145.064317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2154.464095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2154.468216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.476070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2155.489503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2155.489705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2155.490513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2158.927564] Lustre: server umount lustre-OST0000 complete [ 2160.608333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.615224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2160.617357] Lustre: Skipped 1 previous similar message [ 2165.215772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2165.219473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.224493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2166.294922] Lustre: server umount lustre-MDT0000 complete [ 2167.482138] LustreError: 121018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670288 with bad export cookie 5222500350619071132 [ 2167.486012] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2167.487397] LustreError: 121018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2167.599497] Lustre: server umount lustre-MDT0001 complete [ 2171.002076] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2171.931330] Key type lgssc unregistered [ 2172.056369] LNet: 123550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.058864] LNetError: 123550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.066374] LNet: Removed LNI 192.168.203.150@tcp [ 2172.349150] Key type .llcrypt unregistered [ 2172.350127] Key type ._llcrypt unregistered [ 2180.487922] Key type ._llcrypt registered [ 2180.488927] Key type .llcrypt registered [ 2180.524942] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 2186.306580] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2186.694374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2186.702864] alg: No test for adler32 (adler32-zlib) [ 2187.545483] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 2187.623299] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2189.207151] Key type lgssc registered [ 2189.575777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2201.248032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2201.253040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.365429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2202.377716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2202.415082] Lustre: lustre-MDT0000: new disk, initializing [ 2202.441144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2202.447370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2203.727910] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2207.853184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.894826] Lustre: 126887:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2207.922534] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2207.924434] Lustre: Skipped 1 previous similar message [ 2207.965989] Lustre: lustre-MDT0001: new disk, initializing [ 2208.001425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2208.014188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2208.017439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2209.287627] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2212.421062] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2213.613388] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2215.813358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2215.920913] Lustre: lustre-OST0000: new disk, initializing [ 2215.922724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2215.953233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.216454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2217.221407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2217.271780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2217.873722] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2220.912125] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2222.186391] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2222.264354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2223.492243] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2223.556350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2226.256803] Lustre: server umount lustre-OST0000 complete [ 2227.680584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.681053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.685215] Lustre: Skipped 1 previous similar message [ 2232.800189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2232.802126] Lustre: Skipped 2 previous similar messages [ 2236.920992] Lustre: server umount lustre-MDT0000 complete [ 2238.075160] LustreError: 126878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670359 with bad export cookie 9321014209316369548 [ 2238.078936] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2238.081661] LustreError: 126878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2247.703299] Lustre: server umount lustre-MDT0001 complete [ 2250.423922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.571717] LustreError: 130694:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.580787] LustreError: 130694:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2250.593529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.820844] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2254.467377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2255.812277] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2257.561207] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2258.757686] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2261.017133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2261.131675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2261.134429] Lustre: Skipped 1 previous similar message [ 2263.126761] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2264.884552] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2266.182217] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2276.439744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2277.698188] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2277.766497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2281.162541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2281.225495] Lustre: lustre-OST0001: new disk, initializing [ 2281.227434] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2281.258028] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2283.143731] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2286.291776] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2288.638813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2288.642047] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2288.671239] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2289.423338] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2290.754641] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2290.822516] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2292.003367] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cc4c45c1000.ost_server_uuid 50 [ 2292.502173] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9cc4c45c1000.ost_server_uuid in FULL state after 0 sec [ 2298.851791] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2298.853928] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2298.857858] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2298.862076] Lustre: Skipped 1 previous similar message [ 2300.416031] Lustre: server umount lustre-OST0001 complete [ 2303.970395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.971446] LustreError: 134508:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.972207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2303.973894] Lustre: Skipped 2 previous similar messages [ 2303.981139] LustreError: 134508:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2307.748410] Lustre: server umount lustre-OST0000 complete [ 2311.135826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2311.140266] LustreError: Skipped 1 previous similar message [ 2311.141518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.146049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.148776] Lustre: Skipped 1 previous similar message [ 2315.289108] Lustre: server umount lustre-MDT0000 complete [ 2316.487196] LustreError: 133763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670437 with bad export cookie 9321014209316370416 [ 2316.491663] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2316.491753] LustreError: 133763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2316.600678] Lustre: server umount lustre-MDT0001 complete [ 2323.103478] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2324.138480] Key type lgssc unregistered [ 2324.271415] LNet: 136622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.275375] LNetError: 136622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.286384] LNet: Removed LNI 192.168.203.150@tcp [ 2324.586737] Key type .llcrypt unregistered [ 2324.587715] Key type ._llcrypt unregistered [ 2337.747938] Key type ._llcrypt registered [ 2337.749130] Key type .llcrypt registered [ 2337.793434] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 2343.944142] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2344.427341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2344.445316] alg: No test for adler32 (adler32-zlib) [ 2345.309148] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 2345.412894] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2346.999132] Key type lgssc registered [ 2347.402369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2358.458685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2358.463943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.563191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2359.575040] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2359.614692] Lustre: lustre-MDT0000: new disk, initializing [ 2359.640737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2359.648284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2360.933240] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2365.011297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.054145] Lustre: 140711:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2365.077059] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2365.079338] Lustre: Skipped 1 previous similar message [ 2365.117322] Lustre: lustre-MDT0001: new disk, initializing [ 2365.151407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2365.164971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2365.168674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2366.396972] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2369.398247] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2370.587101] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2372.817971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2372.945659] Lustre: lustre-OST0000: new disk, initializing [ 2372.947787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2372.981286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2374.864120] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2377.722363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2377.725661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2377.754577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2377.892174] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2379.849413] Lustre: Failing over lustre-OST0000 [ 2379.917352] Lustre: server umount lustre-OST0000 complete [ 2392.770348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2392.876774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2392.882675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2392.889235] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2394.657305] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2394.695413] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2395.943947] Lustre: Failing over lustre-MDT0000 [ 2396.037372] Lustre: server umount lustre-MDT0000 complete [ 2396.127726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2396.130286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.133605] Lustre: Skipped 1 previous similar message [ 2398.177452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.181229] Lustre: Skipped 1 previous similar message [ 2399.199108] Lustre: 138942:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773670504/real 1773670504] req@00000000ee2a9083 x1859828282581760/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1773670520 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2403.296025] LustreError: 140724:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.301405] LustreError: 140724:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 8 previous similar messages [ 2404.319172] Lustre: 138941:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773670509/real 1773670509] req@000000002b0870eb x1859828282583680/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773670525 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2404.890409] LustreError: 143008:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.416245] LustreError: 141505:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.420970] LustreError: 141505:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2408.890597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.930232] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2408.935593] Lustre: 140716:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.203.150@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2409.010176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2409.029182] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2409.029588] Lustre: lustre-OST0000: Recovery over after 0:15, of 3 clients 3 recovered and 0 were evicted. [ 2409.031780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2409.057297] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2410.010458] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2410.307845] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2414.055606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2414.057543] Lustre: Skipped 1 previous similar message [ 2414.062287] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2414.078156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2419.168989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2419.170505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2419.171050] LustreError: Skipped 1 previous similar message [ 2419.171982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2419.175402] Lustre: Skipped 1 previous similar message [ 2419.179522] Lustre: Skipped 1 previous similar message [ 2421.200345] Lustre: server umount lustre-OST0000 complete [ 2424.288874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2424.289048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.289938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2424.297873] Lustre: Skipped 1 previous similar message [ 2428.564240] Lustre: server umount lustre-MDT0000 complete [ 2429.408373] LustreError: 140724:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.413683] LustreError: 140724:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2429.803204] LustreError: 140704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670551 with bad export cookie 18289661417923862393 [ 2429.807076] LustreError: 140704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2429.807268] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2429.939576] Lustre: server umount lustre-MDT0001 complete [ 2433.559920] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2434.523783] Key type lgssc unregistered [ 2434.641307] LNet: 145937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2434.643791] LNetError: 145937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2434.653280] LNet: Removed LNI 192.168.203.150@tcp [ 2434.950278] Key type .llcrypt unregistered [ 2434.951489] Key type ._llcrypt unregistered [ 2454.764620] Key type ._llcrypt registered [ 2454.766173] Key type .llcrypt registered [ 2454.804375] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2455.245353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2455.251075] alg: No test for adler32 (adler32-zlib) [ 2456.095687] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 2456.175334] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2457.759170] Key type lgssc registered [ 2458.097867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2460.549720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2460.555303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.704069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2462.935613] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2465.916151] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2467.921460] LustreError: 147466:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2467.924828] LustreError: 147466:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2467.927544] LustreError: 147466:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2468.041413] Lustre: server umount lustre-MDT0000 complete [ 2472.741812] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 2475.718677] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2487.344979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.436151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2487.448580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2487.486979] Lustre: lustre-MDT0000: new disk, initializing [ 2487.510558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.516518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2488.785284] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2492.891066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.935143] Lustre: 150614:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2492.960482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2492.962443] Lustre: Skipped 1 previous similar message [ 2493.002286] Lustre: lustre-MDT0001: new disk, initializing [ 2493.037784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2493.048633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2493.051585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2494.334792] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2497.431260] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2498.611784] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2500.887871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2500.995768] Lustre: lustre-OST0000: new disk, initializing [ 2500.998289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2501.029493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2502.872836] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2505.936766] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2508.789553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2508.793584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2508.825762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2513.052222] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2513.603765] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:17:15 (1773670635) [ 2575.327655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2575.327787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2575.328031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2575.336559] Lustre: Skipped 1 previous similar message [ 2578.785516] Lustre: server umount lustre-MDT0000 complete [ 2579.961523] LustreError: 150607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670701 with bad export cookie 8289122354667969110 [ 2579.962473] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2579.966722] LustreError: 150607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2580.447361] LustreError: 150623:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.447409] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2580.454793] LustreError: 150623:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2580.458290] Lustre: Skipped 1 previous similar message [ 2580.458544] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2580.464956] Lustre: Skipped 2 previous similar messages [ 2584.479683] LustreError: 152950:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.479694] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2589.663401] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2589.663401] LustreError: 150621:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.271075] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2594.337749] Lustre: server umount lustre-MDT0001 complete [ 2601.738726] Lustre: server umount lustre-OST0000 complete [ 2604.676371] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 2607.685766] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2619.568632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2619.665446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2619.679625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2619.721747] Lustre: lustre-MDT0000: new disk, initializing [ 2619.748596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2619.755765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2621.097937] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2625.188231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.227158] Lustre: 156832:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2625.249107] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2625.252098] Lustre: Skipped 1 previous similar message [ 2625.294888] Lustre: lustre-MDT0001: new disk, initializing [ 2625.339286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2625.342504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2626.599882] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2629.837875] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2631.110367] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2633.565586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2633.678864] Lustre: lustre-OST0000: new disk, initializing [ 2633.681707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2633.715041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2633.718324] Lustre: Skipped 1 previous similar message [ 2635.224755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2635.233185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2635.288563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2636.173977] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2639.528859] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2641.028491] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2641.097718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2642.517721] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2642.615332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2645.477178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2645.479042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2645.480234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2645.480242] Lustre: Skipped 1 previous similar message [ 2645.486227] Lustre: Skipped 1 previous similar message [ 2645.491916] LustreError: Skipped 1 previous similar message [ 2649.546847] Lustre: server umount lustre-OST0000 complete [ 2651.103909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2651.107088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.712285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2655.712577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.715646] Lustre: Skipped 2 previous similar messages [ 2656.917955] Lustre: server umount lustre-MDT0000 complete [ 2658.184720] LustreError: 156824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670779 with bad export cookie 8289122354668127926 [ 2658.190924] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2658.332630] Lustre: server umount lustre-MDT0001 complete [ 2661.088477] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:19:42 (1773670782) [ 2666.182182] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 2669.747021] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2681.038235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.135642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2681.146742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2681.184778] Lustre: lustre-MDT0000: new disk, initializing [ 2681.207841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2681.215248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2682.513529] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2686.495288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.539627] Lustre: 163142:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2686.642880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2687.900911] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2691.451163] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2692.651120] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2695.112974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2695.330456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2695.388443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2697.671593] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2701.191054] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2705.892295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2705.896913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.896967] LustreError: Skipped 1 previous similar message [ 2705.905928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2705.909876] Lustre: Skipped 1 previous similar message [ 2709.143297] Lustre: server umount lustre-OST0000 complete [ 2716.648467] Lustre: server umount lustre-MDT0000 complete [ 2718.001741] LustreError: 163134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670839 with bad export cookie 8289122354668128794 [ 2718.004903] LustreError: 163134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2718.006871] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2718.132334] Lustre: server umount lustre-MDT0001 complete [ 2720.924190] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:20:42 (1773670842) [ 2726.241507] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 2729.532580] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2744.067880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.178685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2744.190900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2744.193707] Lustre: Skipped 3 previous similar messages [ 2744.230925] Lustre: lustre-MDT0000: new disk, initializing [ 2744.232861] Lustre: Skipped 2 previous similar messages [ 2744.256977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2744.259094] Lustre: Skipped 2 previous similar messages [ 2744.264625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2744.266932] Lustre: Skipped 2 previous similar messages [ 2745.687973] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2750.230171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2750.288256] Lustre: 169057:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2750.423043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2751.778235] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2755.062472] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2756.220213] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2758.897281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2761.040365] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2763.000990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2764.377454] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2768.358126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2768.359281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2768.361128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2768.361134] Lustre: Skipped 4 previous similar messages [ 2768.363384] LustreError: Skipped 1 previous similar message [ 2768.374081] Lustre: Skipped 4 previous similar messages [ 2772.783346] Lustre: server umount lustre-OST0000 complete [ 2780.658656] Lustre: server umount lustre-MDT0000 complete [ 2782.108569] LustreError: 169049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670903 with bad export cookie 8289122354668129865 [ 2782.112576] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2782.114933] LustreError: 169049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2782.247159] Lustre: server umount lustre-MDT0001 complete [ 2786.086449] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2787.467603] Key type lgssc unregistered [ 2787.626611] LNet: 172535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2787.630685] LNetError: 172535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2787.638668] LNet: Removed LNI 192.168.203.150@tcp [ 2788.063166] Key type .llcrypt unregistered [ 2788.065423] Key type ._llcrypt unregistered [ 2797.968793] Key type ._llcrypt registered [ 2797.969815] Key type .llcrypt registered [ 2798.029234] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2798.456937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2798.467659] alg: No test for adler32 (adler32-zlib) [ 2799.353356] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 2799.456709] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2801.071177] Key type lgssc registered [ 2801.581639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2804.586212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2804.595037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.767551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.957898] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2809.610161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2809.728625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2810.905984] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2812.878773] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2813.982428] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2816.774187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2816.966229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.991597] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2820.992723] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2831.339509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2845.665188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2845.668508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2845.672977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2846.690678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2846.691025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2846.695617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.706706] Lustre: Skipped 1 previous similar message [ 2849.246336] Lustre: server umount lustre-OST0000 complete [ 2850.784451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2850.789376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2850.799273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2856.735651] Lustre: server umount lustre-MDT0000 complete [ 2856.929286] LustreError: 173925:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2856.937616] LustreError: 173925:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2858.010228] LustreError: 173906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670979 with bad export cookie 15316933607530331767 [ 2858.014454] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2858.016299] LustreError: 173906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2858.141485] Lustre: server umount lustre-MDT0001 complete [ 2861.553764] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2862.552758] Key type lgssc unregistered [ 2862.681409] LNet: 177124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2862.684344] LNetError: 177124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2862.692344] LNet: Removed LNI 192.168.203.150@tcp [ 2862.987875] Key type .llcrypt unregistered [ 2862.988777] Key type ._llcrypt unregistered [ 2871.632083] Key type ._llcrypt registered [ 2871.633375] Key type .llcrypt registered [ 2871.667432] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2872.074858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2872.081098] alg: No test for adler32 (adler32-zlib) [ 2872.929628] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 2873.011151] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2874.591164] Key type lgssc registered [ 2874.986728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2877.500279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2877.507505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2878.654909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.953760] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2883.135797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.296760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2884.822717] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2887.114769] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2888.580770] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2891.498819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2891.630807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2893.810595] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2895.733944] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2904.935246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2910.177461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2910.178873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2910.179925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2910.184915] Lustre: Skipped 1 previous similar message [ 2911.266519] Lustre: lustre-OST0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 2911.268374] Lustre: Skipped 1 previous similar message [ 2914.448959] Lustre: server umount lustre-OST0000 complete [ 2916.312122] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9cc4c658e000.ost_server_uuid 50 [ 2916.813455] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9cc4c658e000.ost_server_uuid in DISCONN state after 0 sec [ 2917.332441] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2927.286990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2927.287944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2927.298305] Lustre: Skipped 1 previous similar message [ 2930.400429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2930.402207] Lustre: Skipped 2 previous similar messages [ 2933.459970] Lustre: server umount lustre-MDT0000 complete [ 2934.689674] LustreError: 178480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671056 with bad export cookie 4671600738203571940 [ 2934.693369] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2934.823403] Lustre: server umount lustre-MDT0001 complete [ 2937.638957] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:24:19 (1773671059) [ 2939.749609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.902093] LustreError: 182028:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2939.909934] LustreError: 182028:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2939.925763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2941.151570] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2943.956588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.077173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2945.316468] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2947.129250] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2948.374742] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2950.575508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2952.426422] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2954.118097] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2956.300089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2956.379595] Lustre: lustre-OST0001: new disk, initializing [ 2956.381629] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2956.411270] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2956.413861] Lustre: Skipped 1 previous similar message [ 2958.255168] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2958.531919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2958.536726] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2958.570698] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2961.256114] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2963.940025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2963.942799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2963.947617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2968.121515] Lustre: server umount lustre-OST0000 complete [ 2970.014922] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2970.084552] LustreError: 184132:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2970.090869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2970.092212] LustreError: 184132:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2971.354908] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2971.427311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2971.675516] LustreError: 183621:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.201435] LustreError: 184132:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.205970] LustreError: 184132:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2980.323313] LustreError: 183620:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2980.328572] LustreError: 183620:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2984.416349] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2984.416368] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2984.419876] LustreError: Skipped 1 previous similar message [ 2984.423649] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2984.427166] Lustre: Skipped 4 previous similar messages [ 2987.468578] Lustre: server umount lustre-OST0001 complete [ 2990.047673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2990.049795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2990.052895] Lustre: Skipped 1 previous similar message [ 2994.781781] Lustre: server umount lustre-MDT0000 complete [ 2995.680083] LustreError: 182028:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.685327] LustreError: 182028:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2996.001320] LustreError: 182008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671117 with bad export cookie 4671600738203572997 [ 2996.004184] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2996.005464] LustreError: 182008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2996.144900] Lustre: server umount lustre-MDT0001 complete [ 3003.955973] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:25:25 (1773671125) [ 3006.117503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3006.203506] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3006.214661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3006.286580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.515654] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3010.168856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.204683] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3010.216362] Lustre: 187799:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3011.487147] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3013.183762] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3014.425144] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3016.680916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3016.763367] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3018.751741] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3020.108476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3020.506226] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3022.909807] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3022.949597] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3025.254985] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3026.976775] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3028.455530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.459281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3028.459926] Lustre: Skipped 2 previous similar messages [ 3028.462316] Lustre: Skipped 4 previous similar messages [ 3028.519039] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3042.271119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3042.323743] Lustre: server umount lustre-OST0000 complete [ 3042.846086] LustreError: 189941:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.056802] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3054.058070] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3054.060137] LustreError: Skipped 1 previous similar message [ 3059.410630] Lustre: server umount lustre-OST0001 complete [ 3061.728355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3061.730107] Lustre: Skipped 10 previous similar messages [ 3066.784591] Lustre: server umount lustre-MDT0000 complete [ 3067.966318] LustreError: 188527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671189 with bad export cookie 4671600738203574159 [ 3067.969057] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3067.969888] LustreError: 188527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3068.079209] Lustre: server umount lustre-MDT0001 complete [ 3076.874473] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:26:38 (1773671198) [ 3083.091724] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3086.412870] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3099.205393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.293886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3099.305243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3099.348640] Lustre: lustre-MDT0000: new disk, initializing [ 3099.373743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.376740] Lustre: Skipped 3 previous similar messages [ 3099.382413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3100.629822] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3104.735081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.788325] Lustre: 195372:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3104.820160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3104.822856] Lustre: Skipped 1 previous similar message [ 3104.866860] Lustre: lustre-MDT0001: new disk, initializing [ 3104.909774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3104.915738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3106.115635] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3109.177418] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3110.693627] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3113.286315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3113.409666] Lustre: lustre-OST0000: new disk, initializing [ 3113.412411] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3115.241582] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3115.835667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3115.839814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3115.876879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3118.302040] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3119.573721] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3119.646575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3120.888971] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3120.954496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3126.241131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3126.241911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3126.243185] LustreError: Skipped 2 previous similar messages [ 3126.243787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3126.246380] Lustre: Skipped 4 previous similar messages [ 3126.247521] Lustre: Skipped 2 previous similar messages [ 3127.756456] Lustre: server umount lustre-OST0000 complete [ 3135.076763] Lustre: server umount lustre-MDT0000 complete [ 3136.425167] LustreError: 198507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671257 with bad export cookie 4671600738203575895 [ 3136.428337] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3136.428440] LustreError: 198507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3136.543207] Lustre: server umount lustre-MDT0001 complete [ 3139.702215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3139.878433] LustreError: 199181:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3139.886545] LustreError: 199181:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 9 previous similar messages [ 3141.430110] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3145.260162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3147.235524] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3149.639299] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3150.901158] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3153.459203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3155.711643] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3158.091510] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3159.630363] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3165.843063] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 3167.097498] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3167.164797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3170.274667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3170.276769] LustreError: Skipped 1 previous similar message [ 3173.916976] Lustre: server umount lustre-OST0000 complete [ 3176.188159] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3177.890288] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3201.768485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3201.876432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3201.879784] Lustre: Skipped 5 previous similar messages [ 3203.551931] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3205.195185] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3205.688099] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3212.745984] Lustre: server umount lustre-OST0000 complete [ 3214.818872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.824952] Lustre: Skipped 5 previous similar messages [ 3220.116955] Lustre: server umount lustre-MDT0000 complete [ 3221.334982] LustreError: 199164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671342 with bad export cookie 4671600738203576763 [ 3221.337419] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3221.340614] LustreError: 199164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3221.466648] Lustre: server umount lustre-MDT0001 complete [ 3224.291219] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:29:05 (1773671345) [ 3226.465217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3226.614331] LustreError: 204666:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3226.620420] LustreError: 204666:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3227.931093] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3230.640841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.007407] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3233.745746] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3234.927297] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3237.088159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3239.101738] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3240.862403] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3242.249029] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3253.535499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec [ 3254.809550] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3254.877689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3257.016212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3257.089356] Lustre: lustre-OST0001: new disk, initializing [ 3257.090991] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3258.796069] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3261.779850] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3263.038339] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3266.558865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3266.562629] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3266.587219] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3267.188356] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3268.429620] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3268.505055] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3271.655337] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3271.657482] LustreError: Skipped 1 previous similar message [ 3271.658531] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3271.660808] Lustre: Skipped 12 previous similar messages [ 3275.194160] Lustre: server umount lustre-OST0001 complete [ 3276.922367] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3276.993409] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3278.181887] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3278.245543] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3303.651927] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3309.039324] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3309.051928] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3309.059324] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3311.380714] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3321.132887] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3323.764818] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3332.513639] Lustre: server umount lustre-OST0001 complete [ 3355.109277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3355.111904] LustreError: 209575:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.119819] LustreError: Skipped 2 previous similar messages [ 3355.119849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3355.119852] Lustre: Skipped 4 previous similar messages [ 3355.169633] LustreError: 209575:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 26 previous similar messages [ 3360.646780] Lustre: server umount lustre-OST0000 complete [ 3369.458118] Lustre: server umount lustre-MDT0000 complete [ 3371.410396] LustreError: 204648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671492 with bad export cookie 4671600738203577687 [ 3371.416125] LustreError: 204648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3371.416364] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3377.624890] Lustre: server umount lustre-MDT0001 complete [ 3388.244094] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:31:49 (1773671509) [ 3391.508477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3391.717177] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3391.735620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3391.827670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.830260] Lustre: Skipped 5 previous similar messages [ 3393.634195] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3397.508262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3397.554279] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3397.567194] Lustre: 212714:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3399.839685] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3402.790350] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3404.647776] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3408.164721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3408.280567] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3411.421307] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3414.363721] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3420.239181] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3420.320836] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3424.135025] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3427.144498] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3428.045616] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3430.065857] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3430.179346] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3432.187395] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3432.297410] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3434.208477] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cc4c6256800.ost_server_uuid 50 [ 3434.957446] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9cc4c6256800.ost_server_uuid in FULL state after 0 sec [ 3435.486922] Lustre: Permanently deactivating lustre-OST0001 [ 3435.493242] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3436.857377] Lustre: Permanently reactivating lustre-OST0001 [ 3444.820086] Lustre: server umount lustre-OST0001 complete [ 3453.023439] Lustre: server umount lustre-OST0000 complete [ 3460.724984] Lustre: server umount lustre-MDT0000 complete [ 3462.349667] LustreError: 212709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671583 with bad export cookie 4671600738203578961 [ 3462.352966] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3462.355184] LustreError: 212709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3462.507784] Lustre: server umount lustre-MDT0001 complete [ 3474.511229] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:33:15 (1773671595) [ 3478.177379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3478.374757] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3478.405164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3478.417826] Lustre: Skipped 5 previous similar messages [ 3480.395850] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3484.936326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3485.004303] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3485.017083] Lustre: 218814:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3487.762412] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3491.246876] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3493.108600] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3496.622975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3496.776390] Lustre: Permanently deactivating lustre-OST0000 [ 3496.783345] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3499.925532] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3502.986844] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3505.719766] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3505.726664] LustreError: 221252:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 3506.568743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3509.547951] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3512.335865] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3515.505013] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3518.782960] Lustre: Permanently reactivating lustre-OST0000 [ 3518.797371] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3518.800427] Lustre: Skipped 2 previous similar messages [ 3524.627517] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3524.631696] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3524.639070] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3524.647891] Lustre: Skipped 1 previous similar message [ 3524.658796] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3524.677491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3525.604418] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3525.614311] LustreError: Skipped 5 previous similar messages [ 3531.239550] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3531.241487] Lustre: Skipped 21 previous similar messages [ 3531.413343] Lustre: server umount lustre-OST0001 complete [ 3539.672065] Lustre: server umount lustre-OST0000 complete [ 3547.455209] Lustre: server umount lustre-MDT0000 complete [ 3549.311254] LustreError: 218807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671670 with bad export cookie 4671600738203580606 [ 3549.315862] LustreError: 218807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3549.316163] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3549.467730] Lustre: server umount lustre-MDT0001 complete [ 3554.230079] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3555.352698] Key type lgssc unregistered [ 3555.484402] LNet: 223196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3555.487492] LNetError: 223196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3555.498565] LNet: Removed LNI 192.168.203.150@tcp [ 3555.813799] Key type .llcrypt unregistered [ 3555.814891] Key type ._llcrypt unregistered [ 3564.309578] Key type ._llcrypt registered [ 3564.310757] Key type .llcrypt registered [ 3564.348322] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3564.683227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3564.736328] alg: No test for adler32 (adler32-zlib) [ 3565.585355] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 3565.661375] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3567.239134] Key type lgssc registered [ 3567.561431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3570.491113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3570.496028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3571.631574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3572.810991] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3575.408534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3575.443245] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3575.452078] Lustre: 224611:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3575.456220] Lustre: Permanently deactivating lustre-MDT0001 [ 3575.461857] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3575.566280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3576.693183] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3578.270600] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3578.857564] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3578.859577] LustreError: 224652:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3578.865221] LustreError: 224652:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3578.870258] LustreError: 224652:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 3579.311141] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3581.325223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3581.447651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3583.351674] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3585.094980] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3585.570850] LustreError: 226234:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3585.577933] LustreError: 226234:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3587.184645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3587.253654] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3588.894016] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3590.521324] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3592.677339] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3595.752034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3595.752069] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3596.200101] Lustre: Permanently reactivating lustre-MDT0001 [ 3596.206541] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3596.209012] Lustre: Skipped 2 previous similar messages [ 3596.576194] Lustre: lustre-MDT0001: Client f6704a41-0b9e-4a60-856d-35b26d7c77ed (at 192.168.203.50@tcp) reconnecting [ 3598.339249] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3598.858344] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3600.392334] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3600.462825] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3601.702559] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3605.840142] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 4 sec [ 3606.601799] Lustre: Permanently deactivating lustre-MDT0001 [ 3606.606850] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3606.608619] Lustre: Skipped 2 previous similar messages [ 3612.668493] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3621.345486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3621.348062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3621.348881] Lustre: Skipped 2 previous similar messages [ 3621.350701] Lustre: Skipped 1 previous similar message [ 3623.318959] Lustre: server umount lustre-MDT0000 complete [ 3624.528492] LustreError: 224601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671745 with bad export cookie 626681333060322427 [ 3624.530828] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3624.531975] LustreError: 224601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3624.645963] Lustre: server umount lustre-MDT0001 complete [ 3631.928260] Lustre: server umount lustre-OST0000 complete [ 3635.278079] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:35:56 (1773671756) [ 3638.579906] Lustre: server umount lustre-OST0001 complete [ 3640.362037] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3643.053169] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3653.398485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3653.485524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3653.488726] Lustre: Skipped 2 previous similar messages [ 3653.497515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3653.533284] Lustre: lustre-MDT0000: new disk, initializing [ 3653.556162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.562318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3654.784665] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3658.841434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.888821] Lustre: 232290:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3658.919202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3658.922461] Lustre: Skipped 1 previous similar message [ 3658.962194] Lustre: lustre-MDT0001: new disk, initializing [ 3659.010599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3659.014420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3660.271988] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3663.178801] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3664.279710] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3666.381966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3666.481461] Lustre: lustre-OST0000: new disk, initializing [ 3666.483887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3666.509462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3666.512513] Lustre: Skipped 1 previous similar message [ 3668.144679] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3670.387734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3670.392574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3670.426169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3671.090086] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3671.664705] LustreError: 232298:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3675.671835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3675.726567] Lustre: lustre-OST0001: new disk, initializing [ 3675.728182] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3677.410564] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3678.843733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3678.846705] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3678.870248] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3680.216190] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3680.686760] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3681.759207] LustreError: 232298:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3681.765819] LustreError: 235003:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3691.855113] LustreError: 235003:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3694.567789] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3694.568163] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.569731] LustreError: Skipped 1 previous similar message [ 3694.572881] Lustre: Skipped 1 previous similar message [ 3694.574698] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3694.576312] Lustre: Skipped 3 previous similar messages [ 3697.187055] Lustre: lustre-OST0001: Not available for connect from 192.168.203.50@tcp (stopping) [ 3698.749838] Lustre: server umount lustre-OST0001 complete [ 3699.685929] LustreError: 234017:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3699.689908] LustreError: 234017:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3704.808158] LustreError: 234016:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3704.815322] LustreError: 234016:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3709.923726] LustreError: 235695:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3709.927994] LustreError: 235695:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3711.456282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3711.456724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.458440] LustreError: Skipped 1 previous similar message [ 3711.462219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3711.464447] Lustre: Skipped 1 previous similar message [ 3715.047752] LustreError: 234016:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3715.063445] LustreError: 234016:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3716.749146] Lustre: server umount lustre-OST0000 complete [ 3720.160898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3720.161791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3720.166074] Lustre: Skipped 2 previous similar messages [ 3720.168076] Lustre: Skipped 2 previous similar messages [ 3724.185668] Lustre: server umount lustre-MDT0000 complete [ 3725.280298] LustreError: 233085:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3725.286251] LustreError: 233085:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3725.364451] LustreError: 233070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671846 with bad export cookie 626681333060325024 [ 3725.367308] LustreError: 233070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3725.367660] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3731.569580] Lustre: server umount lustre-MDT0001 complete [ 3734.835938] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3735.760843] Key type lgssc unregistered [ 3735.871342] LNet: 236954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3735.873319] LNetError: 236954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3735.880262] LNet: Removed LNI 192.168.203.150@tcp [ 3736.185095] Key type .llcrypt unregistered [ 3736.187158] Key type ._llcrypt unregistered [ 3750.052605] Key type ._llcrypt registered [ 3750.053619] Key type .llcrypt registered [ 3750.094437] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3750.393376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3750.434312] alg: No test for adler32 (adler32-zlib) [ 3751.283360] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 3751.367614] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3752.951181] Key type lgssc registered [ 3753.310445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3755.957491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3755.963421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3757.060328] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3757.067026] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3757.122327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.371321] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3761.082935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.115125] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3761.124991] Lustre: 239121:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3761.213210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3762.496805] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3764.205531] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3765.496799] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3767.847808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3767.937276] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3768.001360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3769.857915] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3771.716283] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3776.043473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3816.932778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3816.933709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3816.935714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3816.937568] LustreError: Skipped 1 previous similar message [ 3816.945242] Lustre: Skipped 1 previous similar message [ 3822.053758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3822.060234] Lustre: Skipped 2 previous similar messages [ 3822.712227] Lustre: server umount lustre-OST0000 complete [ 3824.701824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3830.290237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3830.427612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3832.534836] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3834.651589] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3846.820291] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3846.825296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3846.825440] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3846.831201] Lustre: Skipped 1 previous similar message [ 3872.738628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3872.741758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3872.743862] Lustre: Skipped 1 previous similar message [ 3872.744099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3872.746882] Lustre: Skipped 1 previous similar message [ 3875.004295] Lustre: server umount lustre-OST0000 complete [ 3877.857445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3877.858658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3877.862710] Lustre: Skipped 1 previous similar message [ 3882.977295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3882.981881] Lustre: Skipped 2 previous similar messages [ 3891.167105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3891.232247] Lustre: server umount lustre-MDT0000 complete [ 3892.466040] LustreError: 239844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773672013 with bad export cookie 4335762903820701923 [ 3892.469314] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3892.470546] LustreError: 239844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3892.589644] Lustre: server umount lustre-MDT0001 complete [ 3896.174683] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3897.281442] Key type lgssc unregistered [ 3897.436613] LNet: 244072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3897.439213] LNetError: 244072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3897.447427] LNet: Removed LNI 192.168.203.150@tcp [ 3897.772413] Key type .llcrypt unregistered [ 3897.773576] Key type ._llcrypt unregistered [ 3911.441590] Key type ._llcrypt registered [ 3911.443281] Key type .llcrypt registered [ 3911.511468] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3912.157387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3912.164406] alg: No test for adler32 (adler32-zlib) [ 3913.076233] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 3913.239828] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3914.839181] Key type lgssc registered [ 3915.565253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3920.716951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3920.741061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3922.184495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3926.388421] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3932.646254] LustreError: 245451:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3932.679692] LustreError: 245451:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3935.534316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3935.774811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3940.545634] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3946.137497] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3950.306316] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3956.640812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.859885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3962.973497] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3968.138439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3970.691699] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4002.284448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4002.287345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4002.332976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4004.332290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4004.336233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4007.032836] Lustre: server umount lustre-OST0000 complete [ 4013.024434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4013.037184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.065838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4013.075394] Lustre: Skipped 1 previous similar message [ 4019.682209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4019.692390] Lustre: Skipped 3 previous similar messages [ 4024.806833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4025.311181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4025.711914] Lustre: server umount lustre-MDT0000 complete [ 4028.896719] LustreError: 245451:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4028.913540] LustreError: 245451:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4030.155210] LustreError: 246168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773672151 with bad export cookie 1833759029850299686 [ 4030.169910] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4030.173333] LustreError: 246168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4030.805220] Lustre: server umount lustre-MDT0001 complete [ 4044.832085] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4048.487511] Key type lgssc unregistered [ 4048.815289] LNet: 249407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4048.828657] LNetError: 249407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4048.845262] LNet: Removed LNI 192.168.203.150@tcp [ 4049.994988] Key type .llcrypt unregistered [ 4049.998189] Key type ._llcrypt unregistered [ 4071.388202] Key type ._llcrypt registered [ 4071.399647] Key type .llcrypt registered [ 4071.522844] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4072.918408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4073.279338] alg: No test for adler32 (adler32-zlib) [ 4074.357458] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 4074.577379] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4076.263194] Key type lgssc registered [ 4077.323471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4085.327443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4085.350983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4086.832939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4091.399744] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4096.995354] LustreError: 250612:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4097.010330] LustreError: 250612:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4100.458380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.022782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4105.251680] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4111.899756] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4116.106773] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4123.111142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4123.603659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4129.601280] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4131.840182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4135.460947] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4147.178672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4147.181894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4147.197734] Lustre: Skipped 1 previous similar message [ 4147.198098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4147.208796] Lustre: Skipped 1 previous similar message [ 4148.517431] Lustre: server umount lustre-OST0000 complete [ 4152.290251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4152.310470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4152.315773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4152.337904] Lustre: Skipped 1 previous similar message [ 4157.413784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4157.427134] Lustre: Skipped 1 previous similar message [ 4158.408401] Lustre: server umount lustre-MDT0000 complete [ 4162.315593] LustreError: 252229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773672283 with bad export cookie 13701678791946363375 [ 4162.320266] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4162.326617] LustreError: 252229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4162.869748] Lustre: server umount lustre-MDT0001 complete [ 4174.072422] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4177.154233] Key type lgssc unregistered [ 4177.457763] LNet: 253938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4177.467053] LNetError: 253938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4177.486119] LNet: Removed LNI 192.168.203.150@tcp [ 4178.559073] Key type .llcrypt unregistered [ 4178.563055] Key type ._llcrypt unregistered [ 4203.643071] Key type ._llcrypt registered [ 4203.646123] Key type .llcrypt registered [ 4203.780267] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4205.198224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4205.240479] alg: No test for adler32 (adler32-zlib) [ 4206.322144] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 4206.555450] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4208.207230] Key type lgssc registered [ 4209.385871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4217.904516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4217.928812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.479630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4223.688135] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4229.603815] LustreError: 255337:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4229.632705] LustreError: 255337:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4232.995437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.441305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4237.988317] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4243.613873] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4247.463272] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4254.274134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4254.604414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4260.130121] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4263.879271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4265.550634] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4293.600979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4293.608976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.636219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4294.624029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4294.627805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4294.643893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4295.918437] Lustre: server umount lustre-OST0000 complete [ 4299.745667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.756617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4299.756849] Lustre: Skipped 1 previous similar message [ 4299.775854] Lustre: Skipped 2 previous similar messages [ 4304.868932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4304.881237] Lustre: Skipped 1 previous similar message [ 4305.930110] Lustre: server umount lustre-MDT0000 complete [ 4309.528420] LustreError: 255319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773672430 with bad export cookie 18426548442503941981 [ 4309.535973] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4309.547955] LustreError: 255319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4310.004589] Lustre: server umount lustre-MDT0001 complete [ 4320.389769] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4323.358107] Key type lgssc unregistered [ 4323.741779] LNet: 259291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4323.748193] LNetError: 259291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4323.759485] LNet: Removed LNI 192.168.203.150@tcp [ 4324.815699] Key type .llcrypt unregistered [ 4324.820302] Key type ._llcrypt unregistered [ 4345.110746] Key type ._llcrypt registered [ 4345.123753] Key type .llcrypt registered [ 4345.318454] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4346.736185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4346.986143] alg: No test for adler32 (adler32-zlib) [ 4348.192656] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 4348.503086] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4350.257895] Key type lgssc registered [ 4351.552644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4361.754827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4361.773524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4363.431495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4368.667338] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4372.992668] LustreError: 260512:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4373.025720] LustreError: 260512:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4377.565598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4378.112573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4382.460616] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4387.930578] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4392.243707] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4399.200356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4399.661621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4406.556808] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4407.923249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4412.151960] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4428.257244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4428.277069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4428.305748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4428.776042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4428.782880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4430.475653] Lustre: server umount lustre-OST0000 complete [ 4439.012737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4439.016041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4439.039972] Lustre: Skipped 1 previous similar message [ 4439.056359] Lustre: Skipped 2 previous similar messages [ 4444.128743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4444.133909] Lustre: Skipped 1 previous similar message [ 4449.247450] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4449.250752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4449.281968] Lustre: Skipped 1 previous similar message [ 4449.629687] Lustre: server umount lustre-MDT0000 complete [ 4453.714795] LustreError: 260479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773672575 with bad export cookie 7005660862179695511 [ 4453.732020] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4454.354523] Lustre: server umount lustre-MDT0001 complete [ 4466.469515] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4469.127408] Key type lgssc unregistered [ 4469.400844] LNet: 263869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4469.412236] LNetError: 263869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4469.430857] LNet: Removed LNI 192.168.203.150@tcp [ 4470.580600] Key type .llcrypt unregistered [ 4470.582097] Key type ._llcrypt unregistered [ 4483.210499] Key type ._llcrypt registered [ 4483.212846] Key type .llcrypt registered [ 4483.342512] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4500.799525] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 4517.732527] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4519.068092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4519.178628] alg: No test for adler32 (adler32-zlib) [ 4520.306679] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 4520.554358] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4522.247203] Key type lgssc registered [ 4523.681703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4560.614397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4560.635391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4561.930198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4561.963859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4562.080397] Lustre: lustre-MDT0000: new disk, initializing [ 4562.284115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4562.338787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4566.730261] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4578.688224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4578.863447] Lustre: 267370:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4578.982055] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4578.991231] Lustre: Skipped 1 previous similar message [ 4579.123990] Lustre: lustre-MDT0001: new disk, initializing [ 4579.261481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4579.312888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4579.335756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4584.656818] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4593.640923] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4597.972516] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4606.261783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4606.747344] Lustre: lustre-OST0000: new disk, initializing [ 4606.754763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4606.921486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4614.466617] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4615.260335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4615.268820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4615.407675] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4621.981747] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4626.230590] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4626.500882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4631.241521] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4631.490606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4636.134312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4636.140520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4636.144869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4636.158137] Lustre: Skipped 1 previous similar message [ 4636.183918] Lustre: Skipped 1 previous similar message [ 4639.820978] Lustre: server umount lustre-OST0000 complete [ 4645.864519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4645.876191] LustreError: Skipped 1 previous similar message [ 4645.888601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4645.911665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4649.949988] Lustre: server umount lustre-MDT0000 complete [ 4653.940494] LustreError: 267363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773672775 with bad export cookie 16159284753174271040 [ 4653.959916] LustreError: 267363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4653.964237] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4654.399565] Lustre: server umount lustre-MDT0001 complete [ 4663.206343] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:53:03 (1773672783) [ 4670.079645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4670.663828] LustreError: 271381:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4670.677936] LustreError: 271381:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4670.774221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4675.446298] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4676.068624] LustreError: 271382:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4681.196371] LustreError: 271381:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4684.826833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4685.320165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4690.163721] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4696.552231] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4700.990582] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4707.576514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4707.855300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4713.853878] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4718.148615] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4730.339553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4730.352972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4730.380712] Lustre: Skipped 1 previous similar message [ 4730.404888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4730.414896] Lustre: Skipped 2 previous similar messages [ 4736.500560] LustreError: 272975:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4736.762492] Lustre: server umount lustre-OST0000 complete [ 4746.724590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4746.727854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4746.729529] LustreError: Skipped 1 previous similar message [ 4746.736561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4746.736569] Lustre: Skipped 2 previous similar messages [ 4746.781263] Lustre: Skipped 2 previous similar messages [ 4748.552222] Lustre: server umount lustre-MDT0000 complete [ 4751.842058] LustreError: 271378:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.850908] LustreError: 271378:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4752.510315] LustreError: 271363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773672873 with bad export cookie 16159284753174271922 [ 4752.511878] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4752.526618] LustreError: 271363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4752.802731] Lustre: server umount lustre-MDT0001 complete [ 4764.474539] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4767.253657] Key type lgssc unregistered [ 4767.575670] LNet: 274565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4767.588965] LNetError: 274565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4767.611942] LNet: Removed LNI 192.168.203.150@tcp [ 4768.621855] Key type .llcrypt unregistered [ 4768.624097] Key type ._llcrypt unregistered [ 4795.158860] Key type ._llcrypt registered [ 4795.166197] Key type .llcrypt registered [ 4795.327869] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 4811.598699] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4813.022640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4813.056499] alg: No test for adler32 (adler32-zlib) [ 4814.253632] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 4814.632119] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4816.335339] Key type lgssc registered [ 4817.598677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4862.958823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4862.995336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.330851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4864.386552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4864.529027] Lustre: lustre-MDT0000: new disk, initializing [ 4864.729471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4864.791148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4869.162300] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4880.494342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4880.639829] Lustre: 278359:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4880.720671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4880.727682] Lustre: Skipped 1 previous similar message [ 4880.875762] Lustre: lustre-MDT0001: new disk, initializing [ 4880.989875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4881.046839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4881.061558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4885.536639] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4893.734975] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4897.506168] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4905.334033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4905.851122] Lustre: lustre-OST2710: new disk, initializing [ 4905.858482] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4906.021976] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4914.455751] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4915.880535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4915.900153] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4916.034518] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4922.540369] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4929.420839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4929.637245] Lustre: lustre-OST03e8: new disk, initializing [ 4929.645941] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4929.774827] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4937.164484] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4939.873971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4939.888798] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4940.024613] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4943.091032] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4951.263624] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4951.458148] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4956.791373] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4957.159712] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4962.577926] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4962.912578] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4967.473056] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4967.716083] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4973.025049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4973.036068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4973.049929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4974.745419] Lustre: server umount lustre-MDT0000 complete [ 4976.099491] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4976.132652] Lustre: Skipped 1 previous similar message [ 4978.615745] LustreError: 278350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773673099 with bad export cookie 1093802624699988755 [ 4978.625409] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4978.639586] LustreError: 278350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4979.001902] Lustre: server umount lustre-MDT0001 complete [ 4993.583629] Lustre: server umount lustre-OST2710 complete [ 5006.927607] Lustre: server umount lustre-OST03e8 complete [ 5026.387800] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 5037.266380] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5070.993177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.237315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5071.266671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5071.383299] Lustre: lustre-MDT0000: new disk, initializing [ 5071.516559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5071.544869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5075.841653] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5085.957310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5086.055639] Lustre: 286084:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5086.099581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5086.103807] Lustre: Skipped 1 previous similar message [ 5086.165662] Lustre: lustre-MDT0001: new disk, initializing [ 5086.266665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5086.312379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5086.326830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5091.152053] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5098.865187] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5102.262606] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5108.185470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5108.499180] Lustre: lustre-OST0000: new disk, initializing [ 5108.513189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5108.612812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5114.489815] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5118.496305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5118.512673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5118.611602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5121.766891] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5126.440944] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5126.736111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5131.628073] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5131.820822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5134.309555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5134.311300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5134.322032] Lustre: Skipped 1 previous similar message [ 5134.336254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5139.433125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5139.443459] Lustre: Skipped 1 previous similar message [ 5139.807388] Lustre: server umount lustre-OST0000 complete [ 5144.544628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5144.560719] Lustre: Skipped 1 previous similar message [ 5144.565376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5144.584846] Lustre: Skipped 2 previous similar messages [ 5149.664971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5149.683214] Lustre: Skipped 1 previous similar message [ 5158.367225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5158.566143] Lustre: server umount lustre-MDT0000 complete [ 5159.910888] LustreError: 286097:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5159.940991] LustreError: 286097:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 5162.483783] LustreError: 286077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773673283 with bad export cookie 1093802624699990435 [ 5162.485297] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5162.497688] LustreError: 286077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5162.905395] Lustre: server umount lustre-MDT0001 complete [ 5171.642495] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:01:32 (1773673292) [ 5173.171784] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5174.734883] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:01:35 (1773673295) [ 5177.894664] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5203.687773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5203.940956] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5203.959754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5204.119934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5209.258864] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5217.341383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5217.495486] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5217.555369] Lustre: 291526:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5222.611987] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5228.786106] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5233.159582] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5253.308766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5253.540508] Lustre: Denying initial registration attempt from nid 192.168.203.150@tcp, specified as failover [ 5253.547391] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5253.558635] LustreError: 293140:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 5253.577136] LustreError: 293140:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-OST0000 [ 5253.582463] LustreError: 293140:0:(tgt_mount.c:115:server_deregister_mount()) lustre-OST0000 not registered [ 5253.640751] Lustre: server umount lustre-OST0000 complete [ 5253.644498] LustreError: 293140:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5263.331633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5263.360405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5263.369713] Lustre: Skipped 3 previous similar messages [ 5266.988474] Lustre: server umount lustre-MDT0000 complete [ 5268.448550] LustreError: 292485:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5268.484275] LustreError: 292485:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5270.821883] LustreError: 291520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773673392 with bad export cookie 1093802624699991317 [ 5270.828556] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5270.831789] LustreError: 291520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5271.085299] Lustre: server umount lustre-MDT0001 complete [ 5281.845489] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5284.441243] Key type lgssc unregistered [ 5284.658537] LNet: 294245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5284.663416] LNetError: 294245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5284.675345] LNet: Removed LNI 192.168.203.150@tcp [ 5285.316885] Key type .llcrypt unregistered [ 5285.320215] Key type ._llcrypt unregistered [ 5296.177424] Key type ._llcrypt registered [ 5296.181456] Key type .llcrypt registered [ 5296.293863] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5297.395845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5297.452839] alg: No test for adler32 (adler32-zlib) [ 5298.531813] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 5298.771615] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5300.431135] Key type lgssc registered [ 5301.472849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5358.770797] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5366.883457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5366.901667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.179088] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5368.218586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5368.396372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5373.300075] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5382.309767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5382.403982] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5382.443719] Lustre: 296983:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5382.738838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5387.665404] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5394.082411] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5398.431038] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5418.802259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5419.070231] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5419.321884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5425.626810] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5430.991916] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5435.873719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5435.877876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5435.888100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5436.389185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5436.389791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5436.432915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5436.444635] Lustre: Skipped 1 previous similar message [ 5439.201127] Lustre: server umount lustre-OST0000 complete [ 5444.063937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5444.074652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.090960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5446.624272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5446.636275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.744725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5451.752287] Lustre: Skipped 2 previous similar messages [ 5457.375133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5457.547228] Lustre: server umount lustre-MDT0000 complete [ 5460.863730] LustreError: 296975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773673582 with bad export cookie 2795975322264317351 [ 5460.870135] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5460.876424] LustreError: 296975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5461.269335] Lustre: server umount lustre-MDT0001 complete [ 5471.944423] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5475.906799] Key type lgssc unregistered [ 5476.272216] LNet: 300201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5476.278449] LNetError: 300201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5477.354941] LNet: Removed LNI 192.168.203.150@tcp [ 5478.204806] Key type .llcrypt unregistered [ 5478.208467] Key type ._llcrypt unregistered [ 5502.582583] Key type ._llcrypt registered [ 5502.586264] Key type .llcrypt registered [ 5502.693394] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5504.414382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5504.451709] alg: No test for adler32 (adler32-zlib) [ 5505.585947] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 5505.822048] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5507.503158] Key type lgssc registered [ 5508.604580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5516.846508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5516.871561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5518.519986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5522.970202] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5528.033015] LustreError: 301608:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5528.069266] LustreError: 301608:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5532.285860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5532.650227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5537.194519] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5543.121734] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5546.926171] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5555.450294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5555.907924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5562.316822] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5567.633211] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5578.722177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5578.724917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.729473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5578.747018] Lustre: Skipped 2 previous similar messages [ 5578.764567] Lustre: Skipped 2 previous similar messages [ 5579.347878] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 5581.547631] Lustre: server umount lustre-MDT0000 complete [ 5583.867061] LustreError: 303546:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.881186] LustreError: 303546:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5584.958232] LustreError: 301586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773673706 with bad export cookie 9419146475372594030 [ 5584.969875] LustreError: 301586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5584.974482] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5588.960465] LustreError: 301603:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5588.960972] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5588.970887] LustreError: 301603:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5588.990970] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5591.519165] Lustre: server umount lustre-MDT0001 complete [ 5598.326493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5606.410666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5610.468862] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x82b787d087f4e197 [ 5610.488211] Lustre: MGC192.168.203.150@tcp: Connection restored to 0@lo (at 0@lo) [ 5610.714501] LustreError: 303210:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5610.730540] LustreError: 303210:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5610.792644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5610.842500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 5614.713715] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5614.884822] LustreError: 304862:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5614.909208] LustreError: 304862:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5620.002474] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5622.644739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.121466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5627.494346] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5628.406974] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5633.336201] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5635.054172] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5639.071670] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5640.903519] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5646.817573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5646.827457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5646.848894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5646.860974] Lustre: Skipped 2 previous similar messages [ 5648.864183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5648.873683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5653.984912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5653.998321] Lustre: Skipped 3 previous similar messages [ 5660.133136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5660.297373] Lustre: server umount lustre-OST0000 complete [ 5664.233251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5664.238789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5664.255867] Lustre: Skipped 1 previous similar message [ 5664.266298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5664.269307] Lustre: Skipped 3 previous similar messages [ 5669.840730] Lustre: server umount lustre-MDT0000 complete [ 5673.682813] LustreError: 304052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773673795 with bad export cookie 9419146475372601751 [ 5673.685118] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5673.704914] LustreError: 304052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5674.137876] Lustre: server umount lustre-MDT0001 complete [ 5684.301895] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5687.185351] Key type lgssc unregistered [ 5687.476728] LNet: 307299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5687.482856] LNetError: 307299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5687.512170] LNet: Removed LNI 192.168.203.150@tcp [ 5688.514720] Key type .llcrypt unregistered [ 5688.518650] Key type ._llcrypt unregistered [ 5715.528741] Key type ._llcrypt registered [ 5715.531220] Key type .llcrypt registered [ 5715.635596] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5717.245836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5717.412714] alg: No test for adler32 (adler32-zlib) [ 5718.716234] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 5719.163645] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5720.895207] Key type lgssc registered [ 5721.959466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5729.978127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5729.997148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5731.552908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5736.274346] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5741.024425] LustreError: 308715:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5741.037267] LustreError: 308715:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5745.345207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5745.681891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5749.743856] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5754.926092] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5758.741293] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5766.112740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5766.115331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5766.128526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5766.137372] Lustre: Skipped 1 previous similar message [ 5766.148449] Lustre: Skipped 1 previous similar message [ 5769.906984] Lustre: server umount lustre-MDT0000 complete [ 5770.273883] LustreError: 308715:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.306713] LustreError: 308715:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5773.740245] LustreError: 308697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773673895 with bad export cookie 10083033577801649478 [ 5773.746890] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5773.748585] LustreError: 308697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5774.064350] Lustre: server umount lustre-MDT0001 complete [ 5787.728497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.036827] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5788.074322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5788.269436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.172682] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5801.197836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.284429] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5801.327244] Lustre: 311076:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5801.577510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5805.921276] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5811.049359] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5814.905484] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5824.240661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5824.482059] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5824.493917] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5824.499533] LustreError: 312732:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5824.506364] LustreError: 312732:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-OST0000 [ 5824.513164] LustreError: 312732:0:(tgt_mount.c:115:server_deregister_mount()) lustre-OST0000 not registered [ 5824.573384] Lustre: server umount lustre-OST0000 complete [ 5824.579290] LustreError: 312732:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5834.990600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5835.375485] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5835.692381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5841.726580] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5844.542310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 5847.730856] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5856.378796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5856.465923] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5856.533750] Lustre: lustre-OST0001: new disk, initializing [ 5856.538061] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5856.670393] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5863.016326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5863.033876] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5863.203914] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5863.442988] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5868.712750] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5873.640215] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5873.640920] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5873.643925] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5873.649686] Lustre: Skipped 1 previous similar message [ 5873.658087] LustreError: Skipped 1 previous similar message [ 5877.289581] Lustre: server umount lustre-OST0001 complete [ 5878.765414] LustreError: 313201:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5878.786951] LustreError: 313201:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5883.883563] LustreError: 313200:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.889457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.891656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5883.891662] Lustre: Skipped 1 previous similar message [ 5883.922985] LustreError: 313200:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5883.976138] Lustre: Skipped 1 previous similar message [ 5887.158562] Lustre: server umount lustre-OST0000 complete [ 5893.600328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5893.606721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5893.630132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5893.632548] Lustre: Skipped 1 previous similar message [ 5899.013494] Lustre: server umount lustre-MDT0000 complete [ 5899.233461] LustreError: 311082:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5899.246740] LustreError: 311082:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5902.661227] LustreError: 311066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773674024 with bad export cookie 10083033577801650031 [ 5902.669055] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5902.677377] LustreError: 311066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5903.338149] Lustre: server umount lustre-MDT0001 complete [ 5913.703887] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5916.055551] Key type lgssc unregistered [ 5916.362187] LNet: 315987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5916.376737] LNetError: 315987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5917.414567] LNet: Removed LNI 192.168.203.150@tcp [ 5918.292061] Key type .llcrypt unregistered [ 5918.295605] Key type ._llcrypt unregistered [ 5986.594622] Key type ._llcrypt registered [ 5986.596657] Key type .llcrypt registered [ 5986.702504] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 6002.162739] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6003.275431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6003.468800] alg: No test for adler32 (adler32-zlib) [ 6004.568534] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 6004.787501] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 6006.455161] Key type lgssc registered [ 6007.518170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6042.570892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6042.614856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6043.970602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6043.996703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6044.093518] Lustre: lustre-MDT0000: new disk, initializing [ 6044.159389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6044.184542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6048.890101] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6059.531095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6059.687776] Lustre: 320889:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6059.738476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6059.742644] Lustre: Skipped 1 previous similar message [ 6059.826725] Lustre: lustre-MDT0001: new disk, initializing [ 6059.943392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6059.991371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6060.013617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6063.953814] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6071.648109] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6075.265695] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6081.756852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6082.109572] Lustre: lustre-OST0000: new disk, initializing [ 6082.119661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6082.311796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6087.894808] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6091.350535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6091.371449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6091.512363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6095.727962] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6101.618606] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6102.023232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6106.976914] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6107.161182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6112.229354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6112.230332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6112.252468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6112.268556] Lustre: Skipped 1 previous similar message [ 6115.467103] Lustre: server umount lustre-OST0000 complete [ 6121.441793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6121.449442] LustreError: Skipped 1 previous similar message [ 6121.460031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.482408] Lustre: Skipped 1 previous similar message [ 6121.491329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6122.479919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.519613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6122.522171] Lustre: Skipped 1 previous similar message [ 6127.585908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6127.598594] Lustre: Skipped 1 previous similar message [ 6131.680430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6131.685192] Lustre: Skipped 1 previous similar message [ 6135.263142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6135.366243] Lustre: server umount lustre-MDT0000 complete [ 6139.460530] LustreError: 322627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773674260 with bad export cookie 8158353761395913457 [ 6139.461310] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6139.486492] LustreError: 322627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6139.774280] Lustre: server umount lustre-MDT0001 complete [ 6147.257975] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:17:47 (1773674267) [ 6155.139271] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:17:55 (1773674275) [ 6164.242058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6164.655692] LustreError: 325239:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6164.666989] LustreError: 325239:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 6164.707480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6169.012471] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6170.082105] LustreError: 325240:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6175.202830] LustreError: 325239:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6177.335288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.755516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6182.106952] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6187.463789] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6191.452211] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6198.484782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6199.086949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6205.941316] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6212.313692] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6220.832788] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6222.526835] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6224.321189] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6226.272907] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6228.350497] Lustre: Failing over lustre-MDT0000 [ 6228.629194] Lustre: server umount lustre-MDT0000 complete [ 6228.960916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6228.968692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6228.979467] LustreError: 325239:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.675600] LustreError: 325234:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.700770] LustreError: 325234:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 6245.924696] LustreError: 327011:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6245.948577] LustreError: 327011:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [ 6247.391222] Lustre: 319111:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773674352/real 1773674352] req@000000003ab6be5c x1859832120414080/t0(0) o400->MGC192.168.203.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773674368 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6247.460728] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6249.007950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6257.643409] LustreError: 319108:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000096e5f9b5 x1859832120420096/t0(0) o250->MGC192.168.203.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6257.995223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6258.060620] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6262.315277] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6262.482988] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6263.278329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6263.329116] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6263.400915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6269.155760] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6270.571258] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6274.364771] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6276.754297] Lustre: Failing over lustre-MDT0000 [ 6277.532542] Lustre: server umount lustre-MDT0000 complete [ 6277.723661] LustreError: 325234:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6277.755492] LustreError: 325234:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 15 previous similar messages [ 6278.624033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6278.637202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6278.679497] Lustre: Skipped 4 previous similar messages [ 6293.983299] Lustre: 319112:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773674399/real 1773674399] req@000000009496462c x1859832120433920/t0(0) o400->MGC192.168.203.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773674415 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6294.089579] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6300.286570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6304.143835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6304.226980] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6304.286675] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6309.377328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6309.412748] Lustre: Skipped 2 previous similar messages [ 6309.453302] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6309.520145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6309.904601] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6317.546106] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6319.522515] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6324.706378] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6324.709505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6324.738349] Lustre: Skipped 1 previous similar message [ 6324.775074] Lustre: Skipped 2 previous similar messages [ 6330.712840] Lustre: server umount lustre-MDT0000 complete [ 6334.621807] LustreError: 325219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773674455 with bad export cookie 8158353761395916166 [ 6334.635831] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6334.962305] LustreError: 325236:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6334.984134] LustreError: 325236:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 26 previous similar messages [ 6335.201932] Lustre: server umount lustre-MDT0001 complete [ 6341.735611] Lustre: server umount lustre-OST0000 complete [ 6357.978487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6363.003517] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6372.673069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.088676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6373.091270] Lustre: Skipped 1 previous similar message [ 6377.614336] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6383.136152] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6386.699099] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6392.713690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6399.121570] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6403.459120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6404.536589] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6408.677602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6408.696017] Lustre: Skipped 2 previous similar messages [ 6408.707541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6408.712302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6408.726341] Lustre: Skipped 4 previous similar messages [ 6410.764203] Lustre: server umount lustre-OST0000 complete [ 6420.510886] Lustre: server umount lustre-MDT0000 complete [ 6424.035170] LustreError: 333130:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6424.048781] LustreError: 333130:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 6424.113986] LustreError: 330999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773674545 with bad export cookie 8158353761395916908 [ 6424.116697] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6424.125508] LustreError: 330999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6424.420615] Lustre: server umount lustre-MDT0001 complete [ 6435.229315] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 6437.635854] Key type lgssc unregistered [ 6437.981765] LNet: 334199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6437.990152] LNetError: 334199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6437.997462] LNet: Removed LNI 192.168.203.150@tcp [ 6438.882611] Key type .llcrypt unregistered [ 6438.888271] Key type ._llcrypt unregistered [ 6462.652540] Key type ._llcrypt registered [ 6462.654415] Key type .llcrypt registered [ 6462.809397] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6464.232839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6464.370889] alg: No test for adler32 (adler32-zlib) [ 6465.677495] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 6466.062205] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 6467.858561] Key type lgssc registered [ 6469.079438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6477.385821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6477.411435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6479.044229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6482.559612] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6489.149130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6489.446711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6492.515626] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6497.112958] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6500.263617] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6506.019958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6506.439842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6511.980609] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6517.568783] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6524.850576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6540.257234] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6540.260701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6540.276969] Lustre: Skipped 1 previous similar message [ 6540.285336] Lustre: Skipped 2 previous similar messages [ 6544.729549] Lustre: server umount lustre-MDT0000 complete [ 6545.379926] LustreError: 335681:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6545.403636] LustreError: 335681:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 6547.842396] LustreError: 335661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773674669 with bad export cookie 4965494464851346842 [ 6547.846472] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6547.850252] LustreError: 335661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6548.113867] Lustre: server umount lustre-MDT0001 complete [ 6561.379921] Lustre: server umount lustre-OST0000 complete [ 6573.751701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6574.207561] LustreError: 339072:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6574.277751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6578.409685] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6579.687941] LustreError: 339073:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6583.783926] LustreError: 339072:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6586.494254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6586.793692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6591.013718] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6596.319558] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6599.432870] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6606.827535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6607.237390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6613.587514] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6619.499208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6619.559421] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6632.928934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6632.938400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6632.965364] Lustre: Skipped 1 previous similar message [ 6632.980654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6634.983587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6634.985595] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6634.993471] Lustre: Skipped 1 previous similar message [ 6635.028523] Lustre: Skipped 1 previous similar message [ 6636.310958] Lustre: server umount lustre-MDT0000 complete [ 6639.556954] LustreError: 339053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773674760 with bad export cookie 4965494464851347955 [ 6639.563824] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6639.568163] LustreError: 339053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6639.844195] Lustre: server umount lustre-MDT0001 complete [ 6653.290940] Lustre: server umount lustre-OST0000 complete [ 6666.539188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6667.026550] LustreError: 342649:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6667.144803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6672.088870] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6677.471854] LustreError: 342650:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6677.482197] LustreError: 342650:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6682.440803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6687.123842] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6692.127475] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6695.527089] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6700.752985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6701.021559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6701.026396] Lustre: Skipped 1 previous similar message [ 6706.130933] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6710.583695] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6712.325321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6722.528524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6722.533145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6722.541593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6722.550549] Lustre: Skipped 2 previous similar messages [ 6726.181589] Lustre: server umount lustre-OST0000 complete [ 6732.774540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.781954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6732.793636] Lustre: Skipped 2 previous similar messages [ 6732.803839] Lustre: Skipped 1 previous similar message [ 6735.881699] Lustre: server umount lustre-MDT0000 complete [ 6737.916825] LustreError: 342650:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6737.940993] LustreError: 342650:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6739.391469] LustreError: 345093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773674860 with bad export cookie 4965494464851348977 [ 6739.395577] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6739.404039] LustreError: 345093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6739.802972] Lustre: server umount lustre-MDT0001 complete [ 6749.241437] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 6751.426594] Key type lgssc unregistered [ 6751.703217] LNet: 345828:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6751.711483] LNetError: 345828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6751.731559] LNet: Removed LNI 192.168.203.150@tcp [ 6752.426887] Key type .llcrypt unregistered [ 6752.429371] Key type ._llcrypt unregistered [ 6776.371309] Key type ._llcrypt registered [ 6776.372444] Key type .llcrypt registered [ 6776.463663] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6777.981780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6778.036264] alg: No test for adler32 (adler32-zlib) [ 6779.221839] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 6779.387341] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 6781.035418] Key type lgssc registered [ 6782.347201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6791.757869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6791.780167] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6791.794729] LustreError: 347253:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6791.837172] LustreError: 347253:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6791.849128] LustreError: 347253:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6791.856244] LustreError: 347253:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6791.867973] LustreError: 347253:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6800.837816] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6800.847159] LustreError: 347651:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6800.868662] LustreError: 347651:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6800.878077] LustreError: 347651:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6800.890795] LustreError: 347651:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6800.899493] LustreError: 347651:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6818.706525] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 6822.116951] Key type lgssc unregistered [ 6822.372588] LNet: 348638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6822.379359] LNetError: 348638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6822.390479] LNet: Removed LNI 192.168.203.150@tcp [ 6823.017560] Key type .llcrypt unregistered [ 6823.020365] Key type ._llcrypt unregistered [ 6840.528364] Key type ._llcrypt registered [ 6840.531727] Key type .llcrypt registered [ 6840.619362] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 6854.192593] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6855.528238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6855.669055] alg: No test for adler32 (adler32-zlib) [ 6856.738664] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 6857.022347] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 6858.679183] Key type lgssc registered [ 6859.892383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6891.212543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6891.228967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6892.507194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6892.534636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6892.589207] Lustre: lustre-MDT0000: new disk, initializing [ 6892.656919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6892.677988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6896.113985] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6905.581288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6905.709549] Lustre: 351758:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6905.758695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6905.761882] Lustre: Skipped 1 previous similar message [ 6905.835664] Lustre: lustre-MDT0001: new disk, initializing [ 6905.905939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6905.941904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6905.950173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6909.747722] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6916.192015] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6919.584485] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6925.310935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6925.525818] Lustre: lustre-OST0000: new disk, initializing [ 6925.530644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6925.619039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6930.218954] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6931.490672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6931.497868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6931.566442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6936.893480] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6941.169521] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6941.376887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6945.312663] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6945.541399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6951.913685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6951.916526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6951.932552] LustreError: Skipped 1 previous similar message [ 6951.934290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6951.952949] Lustre: Skipped 1 previous similar message [ 6953.461893] Lustre: server umount lustre-OST0000 complete [ 6957.032898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6957.033582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6957.050055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6957.073891] Lustre: Skipped 2 previous similar messages [ 6962.146071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6962.155091] Lustre: Skipped 2 previous similar messages [ 6962.987280] Lustre: server umount lustre-MDT0000 complete [ 6966.300547] LustreError: 353500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773675087 with bad export cookie 10006100391948867830 [ 6966.302406] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6966.309712] LustreError: 353500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6966.714791] Lustre: server umount lustre-MDT0001 complete [ 6974.278232] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6975.714272] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:31:36 (1773675096) [ 6981.263290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6981.820818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6986.516659] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6987.232559] LustreError: 355814:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6987.256994] LustreError: 355814:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6992.353226] LustreError: 355815:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6995.092609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.329113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6999.505255] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7005.548037] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7009.612707] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7016.336074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7016.724065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7023.382557] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7028.932040] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7036.314656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7036.561980] Lustre: lustre-OST0001: new disk, initializing [ 7036.565936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7036.637185] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7042.700198] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7044.227876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7044.242381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7044.319526] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7051.550650] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7058.509099] Lustre: lustre-OST0001: Not available for connect from 192.168.203.50@tcp (stopping) [ 7059.936255] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7059.946032] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7059.958512] Lustre: Skipped 1 previous similar message [ 7061.127709] Lustre: server umount lustre-OST0001 complete [ 7063.586877] LustreError: 358733:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7070.182424] LustreError: 357411:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7070.210376] LustreError: 357411:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 7075.304100] LustreError: 357413:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7075.329365] LustreError: 357413:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7078.380450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7078.387817] LustreError: Skipped 1 previous similar message [ 7078.392262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7078.434121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7078.436899] Lustre: Skipped 2 previous similar messages [ 7084.169320] Lustre: server umount lustre-OST0000 complete [ 7087.586403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7087.597860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7087.619429] Lustre: Skipped 1 previous similar message [ 7087.627057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7087.630865] Lustre: Skipped 2 previous similar messages [ 7093.640878] Lustre: server umount lustre-MDT0000 complete [ 7095.777601] LustreError: 356526:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7095.787923] LustreError: 356526:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 7096.953768] LustreError: 355794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773675218 with bad export cookie 10006100391948868712 [ 7096.975092] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7103.494266] Lustre: server umount lustre-MDT0001 complete [ 7114.883873] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7117.608242] Key type lgssc unregistered [ 7117.952078] LNet: 360175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7117.957077] LNetError: 360175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7117.969930] LNet: Removed LNI 192.168.203.150@tcp [ 7118.895679] Key type .llcrypt unregistered [ 7118.902263] Key type ._llcrypt unregistered [ 7163.798758] Key type ._llcrypt registered [ 7163.802933] Key type .llcrypt registered [ 7163.914828] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7165.165983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7165.332641] alg: No test for adler32 (adler32-zlib) [ 7166.386701] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 7166.604118] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7168.287194] Key type lgssc registered [ 7169.614578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7177.195440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7177.224892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7178.679827] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7178.730333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7178.987851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7184.091470] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7191.499888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7191.562838] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7191.590720] Lustre: 362936:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7191.825689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7196.268176] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7201.313290] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7206.918551] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7212.514730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7212.528985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7212.551346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7213.043357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7213.081254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7213.093516] Lustre: Skipped 1 previous similar message [ 7215.463075] Lustre: server umount lustre-MDT0000 complete [ 7220.125942] LustreError: 362926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773675341 with bad export cookie 10812625881682760164 [ 7220.139609] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7220.139814] LustreError: 362926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7220.701920] Lustre: server umount lustre-MDT0001 complete [ 7233.046850] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:35:53 (1773675353) [ 7239.330913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7239.752233] LustreError: 365222:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7239.766843] LustreError: 365222:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 7239.830160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7243.782981] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7245.289496] LustreError: 365223:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7250.416096] LustreError: 365222:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7251.949447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7252.334324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7256.510252] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7261.755927] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7265.229975] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7272.150588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7272.335681] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7272.542105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7278.401650] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7283.970549] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7292.072656] Lustre: Permanently deactivating lustre-OST0000 [ 7292.084848] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7301.165143] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7302.673473] LustreError: 367916:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7302.682710] LustreError: 367916:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7302.694911] LustreError: 367916:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7305.545781] LustreError: 367965:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7305.557404] LustreError: 367965:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7313.817210] Lustre: server umount lustre-OST0000 complete [ 7316.669389] LustreError: 368214:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7316.678307] LustreError: 368214:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7319.008271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7319.011774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7319.026592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7325.152463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7325.158689] Lustre: Skipped 3 previous similar messages [ 7330.292562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7330.299333] Lustre: Skipped 1 previous similar message [ 7333.343306] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7333.488630] Lustre: server umount lustre-MDT0000 complete [ 7335.393893] LustreError: 365223:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7335.417822] LustreError: 365223:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7337.000595] LustreError: 365202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773675458 with bad export cookie 10812625881682760752 [ 7337.009256] LustreError: 365202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7337.013195] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7347.408654] Lustre: server umount lustre-MDT0001 complete [ 7356.429530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7360.991076] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7364.029636] LustreError: 369358:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7370.954788] Lustre: 369697:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.150@tcp [ 7373.188901] Lustre: server umount MGS complete [ 7384.698164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.253476] LustreError: 370422:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7385.290711] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7385.298938] Lustre: Skipped 1 previous similar message [ 7385.390686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7390.145431] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7395.811196] LustreError: 370423:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7395.830287] LustreError: 370423:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7398.789645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7399.001113] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7403.797167] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7409.351773] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7413.449120] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7415.217499] Lustre: Permanently reactivating lustre-OST0000 [ 7415.229854] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7415.237975] Lustre: Skipped 2 previous similar messages [ 7442.399281] LustreError: 371976:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7450.556724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7450.842938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7450.850490] Lustre: Skipped 1 previous similar message [ 7456.282827] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7461.523853] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7467.513056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7467.515028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7467.525910] LustreError: Skipped 1 previous similar message [ 7467.528843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7467.576217] Lustre: Skipped 2 previous similar messages [ 7472.398546] Lustre: server umount lustre-OST0000 complete [ 7477.735943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7477.761151] Lustre: Skipped 1 previous similar message [ 7482.850259] LustreError: 371410:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7482.865598] LustreError: 371410:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7482.910057] Lustre: server umount lustre-MDT0000 complete [ 7486.232754] LustreError: 370404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773675607 with bad export cookie 10812625881682761795 [ 7486.236283] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7486.243248] LustreError: 370404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7486.747521] Lustre: server umount lustre-MDT0001 complete [ 7497.241226] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7500.061946] Key type lgssc unregistered [ 7500.315745] LNet: 374478:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7500.323377] LNetError: 374478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7501.348932] LNet: Removed LNI 192.168.203.150@tcp [ 7502.335757] Key type .llcrypt unregistered [ 7502.340814] Key type ._llcrypt unregistered [ 7520.345435] Key type ._llcrypt registered [ 7520.347564] Key type .llcrypt registered [ 7520.460526] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 7534.759090] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7535.622260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7535.657875] alg: No test for adler32 (adler32-zlib) [ 7536.727359] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 7536.925736] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7538.568187] Key type lgssc registered [ 7539.545900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7570.028424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7570.071273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7571.389766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7571.441906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7571.558662] Lustre: lustre-MDT0000: new disk, initializing [ 7571.652561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7571.673840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7576.163451] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7586.321480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7586.461587] Lustre: 377599:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7586.525812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7586.535980] Lustre: Skipped 1 previous similar message [ 7586.626443] Lustre: lustre-MDT0001: new disk, initializing [ 7586.740565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7586.777911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7586.791876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7591.048507] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7598.553726] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7602.273221] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7608.693228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7609.065691] Lustre: lustre-OST0000: new disk, initializing [ 7609.070562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7609.159252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7615.360467] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7617.064077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7617.076865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7617.186218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7622.924660] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7628.201505] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7628.413642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7633.289419] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7633.529594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7637.993030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7638.004541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7638.019240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7638.025959] Lustre: Skipped 1 previous similar message [ 7641.669302] Lustre: server umount lustre-OST0000 complete [ 7648.224360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7648.247912] Lustre: Skipped 1 previous similar message [ 7648.265801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7648.275554] Lustre: Skipped 1 previous similar message [ 7651.559247] Lustre: server umount lustre-MDT0000 complete [ 7655.163466] LustreError: 377594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773675776 with bad export cookie 4639790569006039680 [ 7655.165942] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7655.178275] LustreError: 377594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7655.418795] Lustre: server umount lustre-MDT0001 complete [ 7662.747464] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7664.487118] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:43:04 (1773675784) [ 7670.804048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7671.363041] LustreError: 381655:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7671.379097] LustreError: 381655:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 7671.449838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7675.816797] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7676.914226] LustreError: 381656:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7680.995542] LustreError: 381655:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7684.594570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7685.345880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7690.253413] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7695.286081] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7698.642913] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7704.931750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7705.326646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7711.503430] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7716.642788] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7718.460215] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7720.892415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7726.050652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7726.056365] LustreError: Skipped 1 previous similar message [ 7726.061398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.076426] Lustre: Skipped 1 previous similar message [ 7726.081460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7727.079307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7730.870984] Lustre: server umount lustre-OST0000 complete [ 7736.297462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7736.305189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7736.321727] Lustre: Skipped 1 previous similar message [ 7736.326442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7736.334949] Lustre: Skipped 2 previous similar messages [ 7740.694218] Lustre: server umount lustre-MDT0000 complete [ 7742.439867] LustreError: 381655:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7742.447068] LustreError: 381655:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7743.649569] LustreError: 381636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773675865 with bad export cookie 4639790569006040562 [ 7743.652568] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7743.656776] LustreError: 381636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7743.945904] Lustre: server umount lustre-MDT0001 complete [ 7753.062240] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7755.465656] Key type lgssc unregistered [ 7755.675626] LNet: 385027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7755.680635] LNetError: 385027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7756.720366] LNet: Removed LNI 192.168.203.150@tcp [ 7757.477792] Key type .llcrypt unregistered [ 7757.480585] Key type ._llcrypt unregistered [ 7778.495841] Key type ._llcrypt registered [ 7778.498227] Key type .llcrypt registered [ 7778.580864] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7778.820260] Key type .llcrypt unregistered [ 7778.822817] Key type ._llcrypt unregistered [ 7793.197592] Key type ._llcrypt registered [ 7793.200141] Key type .llcrypt registered [ 7793.271891] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7794.126751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7794.140574] alg: No test for adler32 (adler32-zlib) [ 7795.162518] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 7795.390977] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7797.071228] Key type lgssc registered [ 7798.151818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7805.102244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7805.127762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7806.566953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7810.416330] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7816.160498] LustreError: 387186:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7816.182103] LustreError: 387186:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7817.402423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7817.721861] LustreError: 387187:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7817.736732] LustreError: 387187:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7817.747704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7822.747296] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7822.827302] LustreError: 387186:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7822.850516] LustreError: 387186:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7826.486795] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7827.938407] LustreError: 387186:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7827.946491] LustreError: 387186:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7830.780370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7831.042515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7832.107197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7833.744310] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7841.248575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7841.266914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7841.281740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7842.272495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7842.283826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7842.299482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7842.304659] Lustre: Skipped 1 previous similar message [ 7845.588301] Lustre: server umount lustre-OST0000 complete [ 7848.583167] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7848.588190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7848.598084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7848.793319] Lustre: server umount lustre-MDT0000 complete [ 7851.862516] LustreError: 387168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773675973 with bad export cookie 10790754856858006011 [ 7851.877833] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7852.221974] Lustre: server umount lustre-MDT0001 complete [ 7862.255352] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7864.857622] Key type lgssc unregistered [ 7865.194718] LNet: 390175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7865.204687] LNetError: 390175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7865.225230] LNet: Removed LNI 192.168.203.150@tcp [ 7866.182264] Key type .llcrypt unregistered [ 7866.187839] Key type ._llcrypt unregistered [ 7885.706707] Key type ._llcrypt registered [ 7885.709866] Key type .llcrypt registered [ 7885.818286] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7887.036945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7887.085161] alg: No test for adler32 (adler32-zlib) [ 7888.094568] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 7888.294299] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7889.968666] Key type lgssc registered [ 7891.291118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7898.446074] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7898.483417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7914.016644] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7919.135225] LustreError: 391528:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 7945.823135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7951.181517] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7955.881781] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7961.240275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7961.701930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7961.761481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7965.181541] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7966.696128] LustreError: 391553:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7966.715107] LustreError: 391553:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7971.812806] LustreError: 391554:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7971.894518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7972.239403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7976.129604] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7980.956504] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7983.985598] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7990.757641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7990.761897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7990.771330] Lustre: Skipped 1 previous similar message [ 7990.783895] Lustre: Skipped 1 previous similar message [ 7992.087312] Lustre: server umount lustre-OST0000 complete [ 7995.873152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7995.879168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7995.901659] Lustre: Skipped 1 previous similar message [ 8000.992839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8001.000759] Lustre: Skipped 2 previous similar messages [ 8001.279829] Lustre: server umount lustre-MDT0000 complete [ 8004.188108] LustreError: 391534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773676125 with bad export cookie 284908983757081564 [ 8004.191697] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8004.198928] LustreError: 391534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8004.506474] Lustre: server umount lustre-MDT0001 complete [ 8012.972374] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8015.295626] Key type lgssc unregistered [ 8015.533577] LNet: 394743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8015.544156] LNetError: 394743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8015.560489] LNet: Removed LNI 192.168.203.150@tcp [ 8016.408264] Key type .llcrypt unregistered [ 8016.418493] Key type ._llcrypt unregistered [ 8037.829504] Key type ._llcrypt registered [ 8037.831595] Key type .llcrypt registered [ 8037.962489] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8039.009592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8039.027677] alg: No test for adler32 (adler32-zlib) [ 8040.001532] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 8040.177129] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 8041.807244] Key type lgssc registered [ 8042.796823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8049.422836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8049.438141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8050.931888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8054.859659] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8061.409876] LustreError: 396122:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8061.429068] LustreError: 396122:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 8061.661735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8061.952908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8065.342530] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8069.844664] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8072.359290] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8077.654918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8077.915195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8082.476959] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8086.908024] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8096.371088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8101.861130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8101.863378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8101.875880] Lustre: Skipped 2 previous similar messages [ 8101.878272] Lustre: Skipped 1 previous similar message [ 8102.981753] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 8102.993343] Lustre: Skipped 1 previous similar message [ 8106.455565] Lustre: server umount lustre-MDT0000 complete [ 8106.977036] LustreError: 396122:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8106.984742] LustreError: 396122:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 8108.447513] LustreError: 396101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.50@tcp arrived at 1773676229 with bad export cookie 5252534735829612919 [ 8108.456759] LustreError: 396101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8109.918786] LustreError: 396102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773676231 with bad export cookie 5252534735829612254 [ 8109.924973] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8109.925314] LustreError: 396102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8112.100581] LustreError: 396121:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8112.103825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8112.115167] LustreError: 396121:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8112.132105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8114.148620] LustreError: 396847:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.152726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8114.181016] LustreError: 396847:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8116.137827] Lustre: server umount lustre-OST0000 complete [ 8119.267745] LustreError: 398275:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8119.285148] LustreError: 398275:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8121.886569] Lustre: server umount lustre-MDT0001 complete [ 8130.506211] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8132.593399] Key type lgssc unregistered [ 8132.861701] LNet: 399472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8132.876480] LNetError: 399472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8132.909228] LNet: Removed LNI 192.168.203.150@tcp [ 8133.636417] Key type .llcrypt unregistered [ 8133.638187] Key type ._llcrypt unregistered [ 8154.480938] Key type ._llcrypt registered [ 8154.486563] Key type .llcrypt registered [ 8154.601841] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8155.446901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8155.564371] alg: No test for adler32 (adler32-zlib) [ 8156.726726] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 8156.977923] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 8158.711399] Key type lgssc registered [ 8159.785329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8166.417819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8166.435350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8167.843473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8171.787157] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8178.152403] LustreError: 400874:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8178.181118] LustreError: 400874:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 8179.477835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8179.756515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8183.746910] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8187.727470] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8190.633110] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8196.111149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8196.430374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8201.340448] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8206.445583] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8212.863241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8215.521440] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8215.533901] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8215.558449] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8218.082191] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8218.087387] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8223.207527] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8223.214231] Lustre: Skipped 3 previous similar messages [ 8228.330204] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8228.338154] Lustre: Skipped 2 previous similar messages [ 8229.343208] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8229.547121] Lustre: server umount lustre-MDT0001 complete [ 8233.443587] LustreError: 401894:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8233.460579] LustreError: 401894:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8238.561864] LustreError: 400874:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8238.565034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8238.565502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8238.618251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8238.623924] Lustre: Skipped 1 previous similar message [ 8240.708994] Lustre: server umount lustre-OST0000 complete [ 8243.905584] Lustre: server umount lustre-MDT0000 complete [ 8254.239700] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8256.499155] Key type lgssc unregistered [ 8256.752958] LNet: 404207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8256.759715] LNetError: 404207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8256.792359] LNet: Removed LNI 192.168.203.150@tcp [ 8257.526244] Key type .llcrypt unregistered [ 8257.530667] Key type ._llcrypt unregistered [ 8279.381267] Key type ._llcrypt registered [ 8279.383314] Key type .llcrypt registered [ 8279.468533] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 8291.519534] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8292.519901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8292.571279] alg: No test for adler32 (adler32-zlib) [ 8293.543674] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 8293.713674] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 8295.343236] Key type lgssc registered [ 8296.474959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8326.978669] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8339.250887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8339.273493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8340.514556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8340.541759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8340.616661] Lustre: lustre-MDT0000: new disk, initializing [ 8340.728192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8340.744474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8344.367228] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8357.342222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8357.596604] Lustre: lustre-OST0000: new disk, initializing [ 8357.600971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8357.604511] Lustre: Skipped 1 previous similar message [ 8357.670306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8362.617788] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8366.187644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8366.194885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8366.225371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8368.827983] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8380.985618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8381.096828] Lustre: 408149:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8381.164415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8381.222845] Lustre: lustre-MDT0001: new disk, initializing [ 8381.338046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8381.383635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8381.394482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8384.805443] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8391.951394] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8392.173286] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8395.540330] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8395.735680] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8399.846141] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8399.847691] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8399.860913] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8399.867035] Lustre: Skipped 1 previous similar message [ 8400.951914] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.50@tcp (stopping) [ 8404.961430] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8404.980072] Lustre: Skipped 1 previous similar message [ 8405.957552] Lustre: server umount lustre-MDT0001 complete [ 8415.206799] LustreError: 408158:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8415.219567] LustreError: 408158:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 8420.326477] LustreError: 410363:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8420.334921] LustreError: 410363:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8422.369047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8422.376770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8422.383958] Lustre: Skipped 1 previous similar message [ 8422.388076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8425.441329] LustreError: 408162:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8430.560595] LustreError: 408161:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8430.562339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8430.580158] Lustre: Skipped 1 previous similar message [ 8435.682404] LustreError: 408162:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8436.191101] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8436.370434] Lustre: server umount lustre-OST0000 complete [ 8445.402536] Lustre: server umount lustre-MDT0000 complete [ 8456.683974] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8459.092757] Key type lgssc unregistered [ 8459.405113] LNet: 412716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8459.410254] LNetError: 412716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8459.430108] LNet: Removed LNI 192.168.203.150@tcp [ 8460.113920] Key type .llcrypt unregistered [ 8460.115961] Key type ._llcrypt unregistered [ 8488.304380] Key type ._llcrypt registered [ 8488.306401] Key type .llcrypt registered [ 8488.404305] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8488.660310] Key type .llcrypt unregistered [ 8488.663562] Key type ._llcrypt unregistered [ 8501.378789] Key type ._llcrypt registered [ 8501.385352] Key type .llcrypt registered [ 8501.521480] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8502.304798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8502.350770] alg: No test for adler32 (adler32-zlib) [ 8503.468668] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 8503.725447] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 8505.359199] Key type lgssc registered [ 8506.418240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8526.880910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8526.912554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8528.095902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8528.117647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8528.178631] Lustre: lustre-MDT0000: new disk, initializing [ 8528.238787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8528.254798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8531.337156] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8539.248303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8539.357919] Lustre: 416037:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8539.396818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8539.399866] Lustre: Skipped 1 previous similar message [ 8539.451898] Lustre: lustre-MDT0001: new disk, initializing [ 8539.524507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8539.572101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8539.582383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8542.987335] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8549.391702] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8552.415666] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8557.279207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8557.560551] Lustre: lustre-OST0000: new disk, initializing [ 8557.563781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8557.635224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8561.840575] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8564.783491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8564.794909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8564.912351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8568.150557] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8575.457315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8575.462911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8575.465787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8575.474635] Lustre: Skipped 2 previous similar messages [ 8579.886314] Lustre: server umount lustre-MDT0000 complete [ 8582.412514] LustreError: 416029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773676703 with bad export cookie 17173268686559308542 [ 8582.416297] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8582.423108] LustreError: 416029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8582.620221] Lustre: server umount lustre-MDT0001 complete [ 8595.713925] Lustre: server umount lustre-OST0000 complete [ 8601.873731] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:58:43 (1773676723) [ 8606.518024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8606.825465] LustreError: 419689:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8606.845757] LustreError: 419689:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 8606.886697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8609.323757] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8612.324789] LustreError: 419690:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8614.572453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8614.790928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8617.199447] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8619.963531] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8622.212362] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8639.472328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8639.649388] Lustre: Found index 0 for lustre-OST0000, updating log [ 8639.803344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8643.372539] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8646.616477] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8652.075551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8656.359271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8656.363301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8656.376186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8656.380976] Lustre: Skipped 2 previous similar messages [ 8657.443448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8657.446337] Lustre: Skipped 2 previous similar messages [ 8660.185829] Lustre: server umount lustre-OST0000 complete [ 8676.320412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8676.327235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.334846] Lustre: Skipped 1 previous similar message [ 8676.338453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8678.888262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8680.109805] Lustre: server umount lustre-MDT0000 complete [ 8682.512814] LustreError: 419671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773676803 with bad export cookie 17173268686559309935 [ 8682.517299] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8682.522262] LustreError: 419671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8682.689876] Lustre: server umount lustre-MDT0001 complete [ 8687.957605] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:00:08 (1773676808) [ 8703.391311] LNet: 1 local NIs in recovery (showing 1): 192.168.203.150@tcp [ 9273.206829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9273.355743] LustreError: 425767:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9273.381104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9274.646392] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9277.485634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9277.612735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9278.895628] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9280.676571] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9281.852804] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9297.058218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9297.147215] Lustre: Found index 0 for lustre-OST0000, updating log [ 9297.254351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9299.170049] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9300.327740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 9301.051239] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9302.755856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9302.760570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9302.762868] Lustre: Skipped 2 previous similar messages [ 9303.776789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9304.800867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9307.857020] Lustre: server umount lustre-OST0000 complete [ 9310.176849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9310.177928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9310.181747] Lustre: Skipped 1 previous similar message [ 9310.185055] Lustre: Skipped 2 previous similar messages [ 9314.273254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9314.275916] Lustre: Skipped 7 previous similar messages [ 9315.292113] Lustre: server umount lustre-MDT0000 complete [ 9315.299599] LustreError: 425768:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9315.305719] LustreError: 425768:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 9316.320156] LustreError: 425786:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9316.325690] LustreError: 425786:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 9316.518143] LustreError: 428070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773677437 with bad export cookie 17173268686559311006 [ 9316.520829] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9316.523863] LustreError: 428070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9316.633772] Lustre: server umount lustre-MDT0001 complete [ 9330.079179] LNet: 1 local NIs in recovery (showing 1): 192.168.203.150@tcp [ 9333.870945] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 9336.896453] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9348.575865] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:11:10 (1773677470) [ 9354.269505] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9355.316258] Key type lgssc unregistered [ 9360.543724] LNet: 432030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9360.546387] LNetError: 432030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9426.085341] LNet: Removed LNI 192.168.203.150@tcp [ 9426.358421] Key type .llcrypt unregistered [ 9426.359745] Key type ._llcrypt unregistered [ 9432.738523] Key type ._llcrypt registered [ 9432.740384] Key type .llcrypt registered [ 9432.787812] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9433.099875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9433.124246] alg: No test for adler32 (adler32-zlib) [ 9433.976080] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 9434.054308] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 9435.639167] Key type lgssc registered [ 9436.036613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9437.587429] LNet: Added LNI 192.168.203.150@tcp100 [8/256/0/180] [ 9437.893690] LNet: Added LNI 192.168.203.150@tcp101 [8/256/0/180] [ 9438.191072] LNet: Added LNI 192.168.203.150@tcp102 [8/256/0/180] [ 9438.473414] LNet: Added LNI 192.168.203.150@tcp103 [8/256/0/180] [ 9438.773409] LNet: Added LNI 192.168.203.150@tcp104 [8/256/0/180] [ 9439.068450] LNet: Added LNI 192.168.203.150@tcp105 [8/256/0/180] [ 9439.356699] LNet: Added LNI 192.168.203.150@tcp106 [8/256/0/180] [ 9439.636824] LNet: Added LNI 192.168.203.150@tcp107 [8/256/0/180] [ 9439.912452] LNet: Added LNI 192.168.203.150@tcp108 [8/256/0/180] [ 9440.205373] LNet: Added LNI 192.168.203.150@tcp109 [8/256/0/180] [ 9440.498529] LNet: Added LNI 192.168.203.150@tcp110 [8/256/0/180] [ 9440.789496] LNet: Added LNI 192.168.203.150@tcp111 [8/256/0/180] [ 9441.081050] LNet: Added LNI 192.168.203.150@tcp112 [8/256/0/180] [ 9441.372096] LNet: Added LNI 192.168.203.150@tcp113 [8/256/0/180] [ 9441.669325] LNet: Added LNI 192.168.203.150@tcp114 [8/256/0/180] [ 9441.959763] LNet: Added LNI 192.168.203.150@tcp115 [8/256/0/180] [ 9442.263143] LNet: Added LNI 192.168.203.150@tcp116 [8/256/0/180] [ 9442.557546] LNet: Added LNI 192.168.203.150@tcp117 [8/256/0/180] [ 9442.842982] LNet: Added LNI 192.168.203.150@tcp118 [8/256/0/180] [ 9443.145982] LNet: Added LNI 192.168.203.150@tcp119 [8/256/0/180] [ 9443.464802] LNet: Added LNI 192.168.203.150@tcp120 [8/256/0/180] [ 9443.778709] LNet: Added LNI 192.168.203.150@tcp121 [8/256/0/180] [ 9444.067717] LNet: Added LNI 192.168.203.150@tcp122 [8/256/0/180] [ 9444.355954] LNet: Added LNI 192.168.203.150@tcp123 [8/256/0/180] [ 9444.630576] LNet: Added LNI 192.168.203.150@tcp124 [8/256/0/180] [ 9444.909871] LNet: Added LNI 192.168.203.150@tcp125 [8/256/0/180] [ 9445.185802] LNet: Added LNI 192.168.203.150@tcp126 [8/256/0/180] [ 9445.462164] LNet: Added LNI 192.168.203.150@tcp127 [8/256/0/180] [ 9445.736753] LNet: Added LNI 192.168.203.150@tcp128 [8/256/0/180] [ 9446.016067] LNet: Added LNI 192.168.203.150@tcp129 [8/256/0/180] [ 9446.293917] LNet: Added LNI 192.168.203.150@tcp130 [8/256/0/180] [ 9446.567760] LNet: Added LNI 192.168.203.150@tcp131 [8/256/0/180] [ 9446.845983] LNet: Added LNI 192.168.203.150@tcp132 [8/256/0/180] [ 9447.108261] LNet: Added LNI 192.168.203.150@tcp133 [8/256/0/180] [ 9447.392476] LNet: Added LNI 192.168.203.150@tcp134 [8/256/0/180] [ 9447.682281] LNet: Added LNI 192.168.203.150@tcp135 [8/256/0/180] [ 9449.856621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9449.861499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9450.909282] Lustre: 435010:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [ 9450.974643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9450.985444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9451.016490] Lustre: lustre-MDT0000: new disk, initializing [ 9451.044942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9451.051131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9452.226934] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9456.357802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9456.408163] Lustre: 435027:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9456.436940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9456.439403] Lustre: Skipped 1 previous similar message [ 9456.476026] Lustre: lustre-MDT0001: new disk, initializing [ 9456.490429] Lustre: 435810:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [ 9456.494088] Lustre: 435810:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9456.538074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9456.552485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9456.556730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9457.909704] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9458.996463] Lustre: 436136:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [ 9458.999395] Lustre: 436136:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9461.696791] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9463.041313] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9465.970087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9466.087460] Lustre: lustre-OST0000: new disk, initializing [ 9466.089650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9466.128350] Lustre: 436777:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [ 9466.132245] Lustre: 436777:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 9466.154074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9468.126137] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9471.442281] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9474.026199] Lustre: server umount lustre-OST0000 complete [ 9475.060487] Lustre: 437766:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [ 9475.064757] Lustre: 437766:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9477.087794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9477.089936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9477.095444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9480.160438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9480.160696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9481.502165] Lustre: server umount lustre-MDT0000 complete [ 9482.806712] LustreError: 435809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773677604 with bad export cookie 12478593354746788780 [ 9482.810607] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9482.811657] LustreError: 435809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9482.941533] Lustre: server umount lustre-MDT0001 complete [ 9486.664806] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9487.794270] Key type lgssc unregistered [ 9487.964182] LNet: 438550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9487.968838] LNetError: 438550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9487.971994] LNet: Removed LNI 192.168.203.150@tcp [ 9488.991248] LNet: Removed LNI 192.168.203.150@tcp100 [ 9488.994059] LNetError: 438550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9488.997368] LNetError: 438550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9489.000824] LNet: Removed LNI 192.168.203.150@tcp101 [ 9490.015171] LNet: Removed LNI 192.168.203.150@tcp102 [ 9490.016605] LNetError: 438550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9490.019814] LNetError: 438550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9490.022920] LNet: Removed LNI 192.168.203.150@tcp103 [ 9491.039153] LNet: Removed LNI 192.168.203.150@tcp104 [ 9492.063126] LNet: Removed LNI 192.168.203.150@tcp105 [ 9492.064604] LNetError: 438550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9492.066860] LNetError: 438550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9492.070100] LNet: Removed LNI 192.168.203.150@tcp106 [ 9493.087174] LNet: Removed LNI 192.168.203.150@tcp107 [ 9494.111177] LNet: Removed LNI 192.168.203.150@tcp108 [ 9495.135155] LNet: Removed LNI 192.168.203.150@tcp109 [ 9496.159115] LNet: Removed LNI 192.168.203.150@tcp110 [ 9496.161401] LNetError: 438550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9496.165042] LNetError: 438550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9496.167782] LNet: Removed LNI 192.168.203.150@tcp111 [ 9497.183145] LNet: Removed LNI 192.168.203.150@tcp112 [ 9498.207139] LNet: Removed LNI 192.168.203.150@tcp113 [ 9499.231147] LNet: Removed LNI 192.168.203.150@tcp114 [ 9499.233105] LNet: Removed LNI 192.168.203.150@tcp115 [ 9499.234567] LNet: Removed LNI 192.168.203.150@tcp116 [ 9499.236163] LNet: Removed LNI 192.168.203.150@tcp117 [ 9499.237641] LNet: Removed LNI 192.168.203.150@tcp118 [ 9499.239047] LNet: Removed LNI 192.168.203.150@tcp119 [ 9499.240359] LNet: Removed LNI 192.168.203.150@tcp120 [ 9499.241538] LNet: Removed LNI 192.168.203.150@tcp121 [ 9499.242931] LNet: Removed LNI 192.168.203.150@tcp122 [ 9499.244219] LNet: Removed LNI 192.168.203.150@tcp123 [ 9499.245519] LNet: Removed LNI 192.168.203.150@tcp124 [ 9499.246803] LNet: Removed LNI 192.168.203.150@tcp125 [ 9499.248130] LNet: Removed LNI 192.168.203.150@tcp126 [ 9499.249613] LNet: Removed LNI 192.168.203.150@tcp127 [ 9499.251179] LNet: Removed LNI 192.168.203.150@tcp128 [ 9499.253070] LNet: Removed LNI 192.168.203.150@tcp129 [ 9499.254939] LNet: Removed LNI 192.168.203.150@tcp130 [ 9499.256708] LNet: Removed LNI 192.168.203.150@tcp131 [ 9499.258407] LNet: Removed LNI 192.168.203.150@tcp132 [ 9499.260183] LNet: Removed LNI 192.168.203.150@tcp133 [ 9499.261839] LNet: Removed LNI 192.168.203.150@tcp134 [ 9499.269381] LNet: Removed LNI 192.168.203.150@tcp135 [ 9499.527192] Key type .llcrypt unregistered [ 9499.528374] Key type ._llcrypt unregistered [ 9506.768484] Key type ._llcrypt registered [ 9506.770328] Key type .llcrypt registered [ 9506.809788] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 9512.686411] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9513.050203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9513.074442] alg: No test for adler32 (adler32-zlib) [ 9513.923437] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 9514.010953] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 9515.599150] Key type lgssc registered [ 9515.981724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9527.130729] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:14:08 (1773677648) [ 9532.951969] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9533.954692] Key type lgssc unregistered [ 9534.074334] LNet: 442556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9534.076537] LNetError: 442556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9534.084283] LNet: Removed LNI 192.168.203.150@tcp [ 9534.381418] Key type .llcrypt unregistered [ 9534.382823] Key type ._llcrypt unregistered [ 9541.155794] Key type ._llcrypt registered [ 9541.158138] Key type .llcrypt registered [ 9541.197938] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9541.595444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9541.614113] alg: No test for adler32 (adler32-zlib) [ 9542.478177] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 9542.573468] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 9544.167213] Key type lgssc registered [ 9544.583552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9547.942506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9547.948660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9549.040426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9549.051111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9549.087562] Lustre: lustre-MDT0000: new disk, initializing [ 9549.115876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9549.124576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9550.474257] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9554.808740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9554.857442] Lustre: 443727:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9554.885280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9554.887473] Lustre: Skipped 1 previous similar message [ 9554.926965] Lustre: lustre-MDT0001: new disk, initializing [ 9554.963123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9554.977209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9554.980130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9556.375368] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9559.663650] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9560.869166] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9563.259053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9563.379783] Lustre: lustre-OST0000: new disk, initializing [ 9563.382650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9563.416581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9565.370922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9565.375393] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9565.388514] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9565.411587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9568.586948] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9569.665783] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing lnet_if_list [ 9571.290606] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9571.303981] LNet: Added LNI 192.168.203.150@tcp10 [8/256/0/180] [ 9573.156875] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9573.169708] LNet: Added LNI 192.168.203.150@tcp11 [8/256/0/180] [ 9574.559698] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9574.572485] LNet: Added LNI 192.168.203.150@tcp12 [8/256/0/180] [ 9576.030951] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9576.044281] LNet: Added LNI 192.168.203.150@tcp13 [8/256/0/180] [ 9577.529492] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9577.544602] LNet: Added LNI 192.168.203.150@tcp14 [8/256/0/180] [ 9579.091104] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9579.102248] LNet: Added LNI 192.168.203.150@tcp15 [8/256/0/180] [ 9580.611760] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9580.623745] LNet: Added LNI 192.168.203.150@tcp16 [8/256/0/180] [ 9582.178850] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 9582.192834] LNet: Added LNI 192.168.203.150@tcp17 [8/256/0/180] [ 9583.898653] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 9583.911826] LNet: Added LNI 192.168.203.150@tcp18 [8/256/0/180] [ 9585.574030] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 9585.587919] LNet: Added LNI 192.168.203.150@tcp19 [8/256/0/180] [ 9587.263587] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 9587.275392] LNet: Added LNI 192.168.203.150@tcp20 [8/256/0/180] [ 9589.055739] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 9589.068427] LNet: Added LNI 192.168.203.150@tcp21 [8/256/0/180] [ 9590.827927] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 9590.840209] LNet: Added LNI 192.168.203.150@tcp22 [8/256/0/180] [ 9592.629764] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 9592.640384] LNet: Added LNI 192.168.203.150@tcp23 [8/256/0/180] [ 9594.394564] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 9594.407272] LNet: Added LNI 192.168.203.150@tcp24 [8/256/0/180] [ 9596.217193] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 9596.227810] LNet: Added LNI 192.168.203.150@tcp25 [8/256/0/180] [ 9598.041633] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 9598.054366] LNet: Added LNI 192.168.203.150@tcp26 [8/256/0/180] [ 9599.933400] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 9599.946508] LNet: Added LNI 192.168.203.150@tcp27 [8/256/0/180] [ 9601.921510] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 9601.933153] LNet: Added LNI 192.168.203.150@tcp28 [8/256/0/180] [ 9603.917618] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 9603.932270] LNet: Added LNI 192.168.203.150@tcp29 [8/256/0/180] [ 9605.892812] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 9605.904605] LNet: Added LNI 192.168.203.150@tcp30 [8/256/0/180] [ 9607.880805] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 9607.893536] LNet: Added LNI 192.168.203.150@tcp31 [8/256/0/180] [ 9609.886739] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 9609.899085] LNet: Added LNI 192.168.203.150@tcp32 [8/256/0/180] [ 9612.058356] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 9612.071696] LNet: Added LNI 192.168.203.150@tcp33 [8/256/0/180] [ 9614.311643] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 9614.326556] LNet: Added LNI 192.168.203.150@tcp34 [8/256/0/180] [ 9616.639554] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 9616.652533] LNet: Added LNI 192.168.203.150@tcp35 [8/256/0/180] [ 9619.052882] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 9619.066868] LNet: Added LNI 192.168.203.150@tcp36 [8/256/0/180] [ 9621.349733] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 9621.362661] LNet: Added LNI 192.168.203.150@tcp37 [8/256/0/180] [ 9623.599385] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 9623.615852] LNet: Added LNI 192.168.203.150@tcp38 [8/256/0/180] [ 9626.040736] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 9626.054466] LNet: Added LNI 192.168.203.150@tcp39 [8/256/0/180] [ 9628.528304] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 9628.539353] LNet: Added LNI 192.168.203.150@tcp40 [8/256/0/180] [ 9631.039822] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 9631.051767] LNet: Added LNI 192.168.203.150@tcp41 [8/256/0/180] [ 9633.561252] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 9633.576227] LNet: Added LNI 192.168.203.150@tcp42 [8/256/0/180] [ 9636.167407] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 9636.181731] LNet: Added LNI 192.168.203.150@tcp43 [8/256/0/180] [ 9638.763601] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 9638.776363] LNet: Added LNI 192.168.203.150@tcp44 [8/256/0/180] [ 9641.425032] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 9641.436175] LNet: Added LNI 192.168.203.150@tcp45 [8/256/0/180] [ 9643.948365] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 9643.961221] LNet: Added LNI 192.168.203.150@tcp46 [8/256/0/180] [ 9646.531165] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 9646.542790] LNet: Added LNI 192.168.203.150@tcp47 [8/256/0/180] [ 9649.406390] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 9649.418300] LNet: Added LNI 192.168.203.150@tcp48 [8/256/0/180] [ 9652.346775] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 9652.360747] LNet: Added LNI 192.168.203.150@tcp49 [8/256/0/180] [ 9654.967893] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 9654.981859] LNet: Added LNI 192.168.203.150@tcp50 [8/256/0/180] [ 9662.431752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9662.435258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9662.441422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9662.944613] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9662.945802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9662.949283] Lustre: Skipped 1 previous similar message [ 9668.065687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9668.067695] Lustre: Skipped 2 previous similar messages [ 9668.445208] Lustre: server umount lustre-MDT0000 complete [ 9669.706157] LustreError: 443719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773677791 with bad export cookie 12328298582540510225 [ 9669.708706] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9669.713432] LustreError: 443719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9669.831886] Lustre: server umount lustre-MDT0001 complete [ 9677.351564] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9680.422723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9689.567272] Lustre: 443004:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773677794/real 1773677794] req@000000000bb33fa2 x1859835829239680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773677810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9689.577115] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9690.591168] Lustre: 443001:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773677795/real 1773677795] req@0000000041dc1ff0 x1859835829239808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773677811 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9693.663149] Lustre: 443001:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773677799/real 1773677799] req@000000008391a7e8 x1859835829239936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773677815 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9694.687301] LustreError: 443000:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005fd94045 x1859835829240704/t0(0) o250->MGC192.168.203.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9694.808091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9695.711094] Lustre: 443004:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773677801/real 1773677801] req@00000000eb4148c5 x1859835829240192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773677817 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9696.173375] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9699.204210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9699.334367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9700.679357] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9701.343105] Lustre: 443004:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773677806/real 1773677806] req@00000000a7b8174a x1859835829240320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773677822 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9702.552705] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9703.897306] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9704.421169] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9714.657512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9714.658332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9714.659762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9714.667462] Lustre: Skipped 1 previous similar message [ 9717.461789] Lustre: server umount lustre-OST0000 complete [ 9719.776761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9719.776990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9719.777604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9719.777608] Lustre: Skipped 1 previous similar message [ 9719.780598] Lustre: Skipped 1 previous similar message [ 9730.016071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9730.018862] Lustre: Skipped 4 previous similar messages [ 9733.087110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9733.153059] Lustre: server umount lustre-MDT0000 complete [ 9734.451136] LustreError: 445467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773677855 with bad export cookie 12328298582540512311 [ 9734.453966] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9734.454464] LustreError: 445467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9734.568703] Lustre: server umount lustre-MDT0001 complete [ 9738.178552] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9739.256085] Key type lgssc unregistered [ 9739.423439] LNet: 471394:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9739.427408] LNetError: 471394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9739.430570] LNet: Removed LNI 192.168.203.150@tcp [ 9739.432413] LNet: Removed LNI 192.168.203.150@tcp10 [ 9739.434265] LNet: Removed LNI 192.168.203.150@tcp11 [ 9739.435729] LNet: Removed LNI 192.168.203.150@tcp12 [ 9739.437021] LNet: Removed LNI 192.168.203.150@tcp13 [ 9739.438307] LNet: Removed LNI 192.168.203.150@tcp14 [ 9739.439819] LNet: Removed LNI 192.168.203.150@tcp15 [ 9739.441276] LNet: Removed LNI 192.168.203.150@tcp16 [ 9739.443410] LNet: Removed LNI 192.168.203.150@tcp17 [ 9739.445281] LNet: Removed LNI 192.168.203.150@tcp18 [ 9739.446354] LNet: Removed LNI 192.168.203.150@tcp19 [ 9739.447928] LNet: Removed LNI 192.168.203.150@tcp20 [ 9739.449596] LNet: Removed LNI 192.168.203.150@tcp21 [ 9739.451012] LNet: Removed LNI 192.168.203.150@tcp22 [ 9739.452266] LNet: Removed LNI 192.168.203.150@tcp23 [ 9739.453919] LNet: Removed LNI 192.168.203.150@tcp24 [ 9739.455443] LNet: Removed LNI 192.168.203.150@tcp25 [ 9739.456867] LNet: Removed LNI 192.168.203.150@tcp26 [ 9739.458213] LNet: Removed LNI 192.168.203.150@tcp27 [ 9739.459407] LNet: Removed LNI 192.168.203.150@tcp28 [ 9739.460536] LNet: Removed LNI 192.168.203.150@tcp29 [ 9739.462073] LNet: Removed LNI 192.168.203.150@tcp30 [ 9739.463845] LNet: Removed LNI 192.168.203.150@tcp31 [ 9739.465554] LNet: Removed LNI 192.168.203.150@tcp32 [ 9739.466765] LNet: Removed LNI 192.168.203.150@tcp33 [ 9739.468133] LNet: Removed LNI 192.168.203.150@tcp34 [ 9739.469272] LNet: Removed LNI 192.168.203.150@tcp35 [ 9739.470429] LNet: Removed LNI 192.168.203.150@tcp36 [ 9739.471988] LNet: Removed LNI 192.168.203.150@tcp37 [ 9739.473406] LNet: Removed LNI 192.168.203.150@tcp38 [ 9739.474921] LNet: Removed LNI 192.168.203.150@tcp39 [ 9739.476378] LNet: Removed LNI 192.168.203.150@tcp40 [ 9739.477415] LNet: Removed LNI 192.168.203.150@tcp41 [ 9739.478928] LNet: Removed LNI 192.168.203.150@tcp42 [ 9739.480180] LNet: Removed LNI 192.168.203.150@tcp43 [ 9739.481206] LNet: Removed LNI 192.168.203.150@tcp44 [ 9739.482116] LNet: Removed LNI 192.168.203.150@tcp45 [ 9739.483178] LNet: Removed LNI 192.168.203.150@tcp46 [ 9739.484329] LNet: Removed LNI 192.168.203.150@tcp47 [ 9739.485326] LNet: Removed LNI 192.168.203.150@tcp48 [ 9739.486351] LNet: Removed LNI 192.168.203.150@tcp49 [ 9739.491894] LNet: Removed LNI 192.168.203.150@tcp50 [ 9739.764843] Key type .llcrypt unregistered [ 9739.766277] Key type ._llcrypt unregistered [ 9747.399498] Key type ._llcrypt registered [ 9747.400834] Key type .llcrypt registered [ 9747.441560] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9747.868719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9747.887765] alg: No test for adler32 (adler32-zlib) [ 9748.743852] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 9748.825883] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 9750.407199] Key type lgssc registered [ 9750.805084] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9753.436043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9753.444542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9754.600433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9755.914704] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9758.745354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9758.905121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9760.242293] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9762.065875] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9763.342031] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9765.794386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9765.943264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9767.952235] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9769.843602] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9779.237459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9783.983581] loop0: detected capacity change from 0 to 204800000 [ 9784.083399] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9784.087918] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9784.092879] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9786.119246] loop0: detected capacity change from 0 to 204800000 [ 9786.178517] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9786.184036] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9786.188714] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9787.928303] loop0: detected capacity change from 0 to 204800000 [ 9789.818084] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9789.866532] Lustre: fs15246-OST0000: new disk, initializing [ 9789.868588] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 9789.886285] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9791.750287] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9795.565149] loop1: detected capacity change from 0 to 204800000 [ 9797.462256] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9797.500446] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 9797.522482] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 9797.566579] Lustre: fs15246-MDT0000: new disk, initializing [ 9797.621740] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9797.631309] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9797.672607] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9798.687786] Lustre: 476997:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000ada767b9 x1859836046244096/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773677936 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 9798.904938] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9807.775207] LustreError: 476997:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 9807.778571] LustreError: 476997:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 9807.781919] LustreError: 476997:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 9808.868143] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9808.872915] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 9812.695909] Lustre: server umount fs15246-MDT0000 complete [ 9813.986744] LustreError: 474769:0:(ldlm_lib.c:1178:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9813.993648] LustreError: 474769:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 9815.997555] Lustre: server umount fs15246-OST0000 complete [ 9820.128518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9820.128671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9820.132438] LustreError: Skipped 1 previous similar message [ 9820.136481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9820.143129] Lustre: Skipped 1 previous similar message [ 9824.225250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9824.228282] Lustre: Skipped 1 previous similar message [ 9825.552106] Lustre: server umount lustre-OST0000 complete [ 9829.344384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9829.344978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9829.348396] Lustre: Skipped 2 previous similar messages [ 9834.465025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9834.467404] Lustre: Skipped 2 previous similar messages [ 9841.119125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9841.178740] Lustre: server umount lustre-MDT0000 complete [ 9842.299456] LustreError: 472599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773677963 with bad export cookie 7943649201109203908 [ 9842.302351] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9842.302516] LustreError: 472599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9842.419058] Lustre: server umount lustre-MDT0001 complete [ 9845.759273] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9846.759896] Key type lgssc unregistered [ 9846.876397] LNet: 480060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9846.879621] LNetError: 480060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9846.888336] LNet: Removed LNI 192.168.203.150@tcp [ 9847.178532] Key type .llcrypt unregistered [ 9847.180279] Key type ._llcrypt unregistered [ 9854.185312] Key type ._llcrypt registered [ 9854.186864] Key type .llcrypt registered [ 9854.225201] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9854.528036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9854.533659] alg: No test for adler32 (adler32-zlib) [ 9855.387483] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 9855.471540] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 9857.055145] Key type lgssc registered [ 9857.433625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9869.289188] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 9872.212246] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9883.516755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9883.524782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9884.604622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9884.615613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9884.650853] Lustre: lustre-MDT0000: new disk, initializing [ 9884.674831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9884.681411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9885.873252] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9889.833313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9889.872808] Lustre: 484459:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9889.892693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9889.894626] Lustre: Skipped 1 previous similar message [ 9889.927286] Lustre: lustre-MDT0001: new disk, initializing [ 9889.957642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9889.968831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9889.972353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9891.135866] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9894.027180] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9895.103271] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9897.217883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9897.309316] Lustre: lustre-OST0000: new disk, initializing [ 9897.310951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9897.333714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9898.990498] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9900.149628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9900.153564] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9900.180097] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9901.889942] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9903.138857] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9903.207553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9904.446087] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9904.511178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9905.633765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9905.634504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9905.635672] LustreError: Skipped 1 previous similar message [ 9905.638959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9905.641643] Lustre: Skipped 1 previous similar message [ 9910.753692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9910.756020] Lustre: Skipped 1 previous similar message [ 9911.181152] Lustre: server umount lustre-OST0000 complete [ 9915.873092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9915.873121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9915.880981] Lustre: Skipped 1 previous similar message [ 9915.886711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9915.888800] Lustre: Skipped 1 previous similar message [ 9918.488530] Lustre: server umount lustre-MDT0000 complete [ 9919.704169] LustreError: 484451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773678041 with bad export cookie 12656266051670983426 [ 9919.706323] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9919.708345] LustreError: 484451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9919.814233] Lustre: server umount lustre-MDT0001 complete [ 9927.478709] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 9930.383037] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9939.838933] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9821 sec ========== 12:21:01 (1773678061) [ 9940.346894] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:21:01 (1773678061) === [ 9941.357263] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:21:02 (1773678062) === [ 9953.583951] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9954.493716] Key type lgssc unregistered [ 9954.616275] LNet: 492630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9954.618270] LNetError: 492630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9954.624256] LNet: Removed LNI 192.168.203.150@tcp [ 9954.869922] Key type .llcrypt unregistered [ 9954.870800] Key type ._llcrypt unregistered