[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488818851 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003252] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008430] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011141] LSM: Security Framework initializing [ 0.012060] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.014087] *** VALIDATE selinux *** [ 0.022344] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027723] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030103] *** VALIDATE tmpfs *** [ 0.032114] *** VALIDATE proc *** [ 0.033245] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035247] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042500] debug: unmapping init [mem 0xffffffff9aa59000-0xffffffff9aa60fff] [ 0.044959] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045668] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053268] rcu: Hierarchical SRCU implementation. [ 0.055367] smp: Bringing up secondary CPUs ... [ 0.056558] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.061173] smp: Brought up 1 node, 4 CPUs [ 0.063019] smpboot: Max logical packages: 1 [ 0.064013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.155798] node 0 deferred pages initialised in 87ms [ 0.159010] devtmpfs: initialized [ 0.160215] x86/mm: Memory block size: 128MB [ 0.162705] gcov: version magic: 0x41383552 [ 0.164234] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166226] pinctrl core: initialized pinctrl subsystem [ 0.167175] [ 0.167724] ************************************************************* [ 0.168011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169011] ** ** [ 0.170012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171014] ** ** [ 0.172015] ** This means that this kernel is built to expose internal ** [ 0.173013] ** IOMMU data structures, which may compromise security on ** [ 0.174012] ** your system. ** [ 0.175012] ** ** [ 0.176011] ** If you see this message and you are not debugging the ** [ 0.177014] ** kernel, report this immediately to your vendor! ** [ 0.178012] ** ** [ 0.179009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180009] ************************************************************* [ 0.181663] NET: Registered protocol family 16 [ 0.182305] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.185422] cpuidle: using governor menu [ 0.187704] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.190560] PCI: Using configuration type 1 for base access [ 0.192120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.201014] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.202022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.203152] cryptd: max_cpu_qlen set to 1000 [ 0.206052] ACPI: Added _OSI(Module Device) [ 0.207015] ACPI: Added _OSI(Processor Device) [ 0.209009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.210009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214387] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220359] ACPI: Interpreter enabled [ 0.222062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223011] ACPI: Using IOAPIC for interrupt routing [ 0.224114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.228425] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.238305] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.240026] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.242014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244060] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249240] acpiphp: Slot [2] registered [ 0.251125] acpiphp: Slot [5] registered [ 0.253097] acpiphp: Slot [6] registered [ 0.254107] acpiphp: Slot [7] registered [ 0.256115] acpiphp: Slot [8] registered [ 0.257117] acpiphp: Slot [9] registered [ 0.259110] acpiphp: Slot [10] registered [ 0.260113] acpiphp: Slot [3] registered [ 0.262094] acpiphp: Slot [4] registered [ 0.263078] acpiphp: Slot [11] registered [ 0.264105] acpiphp: Slot [12] registered [ 0.266098] acpiphp: Slot [13] registered [ 0.268100] acpiphp: Slot [14] registered [ 0.269116] acpiphp: Slot [15] registered [ 0.271106] acpiphp: Slot [16] registered [ 0.273134] acpiphp: Slot [17] registered [ 0.274101] acpiphp: Slot [18] registered [ 0.276110] acpiphp: Slot [19] registered [ 0.277093] acpiphp: Slot [20] registered [ 0.279110] acpiphp: Slot [21] registered [ 0.280069] acpiphp: Slot [22] registered [ 0.281091] acpiphp: Slot [23] registered [ 0.283098] acpiphp: Slot [24] registered [ 0.284095] acpiphp: Slot [25] registered [ 0.286091] acpiphp: Slot [26] registered [ 0.287083] acpiphp: Slot [27] registered [ 0.288082] acpiphp: Slot [28] registered [ 0.290089] acpiphp: Slot [29] registered [ 0.291119] acpiphp: Slot [30] registered [ 0.292092] acpiphp: Slot [31] registered [ 0.294052] PCI host bridge to bus 0000:00 [ 0.295016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.297019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.299019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.302021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.304027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.307025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.309189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.311900] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.315238] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.324016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.329109] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.331016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.333016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.336020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.339035] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.341774] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.344041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.346773] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.351013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.361823] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.366014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.372528] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.379012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.390017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.404014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.411388] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.418015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.424015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.441015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.453179] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.457012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.462013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.475014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.486671] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.492979] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.496979] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.512020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.522690] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.529013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.536013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.551014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.559427] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.569013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.573014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.585013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.593782] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.596380] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.600291] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.603568] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.605227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.610131] iommu: Default domain type: Passthrough [ 0.612447] SCSI subsystem initialized [ 0.613139] ACPI: bus type USB registered [ 0.615121] usbcore: registered new interface driver usbfs [ 0.616062] usbcore: registered new interface driver hub [ 0.618098] usbcore: registered new device driver usb [ 0.619128] pps_core: LinuxPPS API ver. 1 registered [ 0.621011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.623054] PTP clock support registered [ 0.625087] EDAC MC: Ver: 3.0.0 [ 0.626157] PCI: Using ACPI for IRQ routing [ 0.629379] NetLabel: Initializing [ 0.631011] NetLabel: domain hash size = 128 [ 0.632009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.634078] NetLabel: unlabeled traffic allowed by default [ 0.636144] vgaarb: loaded [ 0.638217] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.640015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.644322] clocksource: Switched to clocksource kvm-clock [ 0.749546] VFS: Disk quotas dquot_6.6.0 [ 0.751087] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.753708] *** VALIDATE ramfs *** [ 0.755049] *** VALIDATE hugetlbfs *** [ 0.756749] pnp: PnP ACPI init [ 0.758714] pnp: PnP ACPI: found 6 devices [ 0.782972] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.786477] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.788884] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.791196] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.793720] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.796308] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.799115] NET: Registered protocol family 2 [ 0.801655] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.805430] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.809041] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.814154] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.818557] TCP: Hash tables configured (established 65536 bind 65536) [ 0.821975] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.825686] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.828767] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.830900] NET: Registered protocol family 1 [ 0.833419] RPC: Registered named UNIX socket transport module. [ 0.835754] RPC: Registered udp transport module. [ 0.837530] RPC: Registered tcp transport module. [ 0.839446] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.842066] NET: Registered protocol family 44 [ 0.843454] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.845556] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.847880] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.850420] PCI: CLS 0 bytes, default 64 [ 0.852508] Unpacking initramfs... [ 2.235697] debug: unmapping init [mem 0xffff978cbcc54000-0xffff978cbffbffff] [ 2.239534] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.242178] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.245467] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.734864] Initialise system trusted keyrings [ 2.736669] Key type blacklist registered [ 2.739299] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.747858] zbud: loaded [ 2.750876] *** VALIDATE nfs *** [ 2.752086] *** VALIDATE nfs4 *** [ 2.753664] pstore: using deflate compression [ 2.758344] Platform Keyring initialized [ 2.850537] NET: Registered protocol family 38 [ 2.852195] Key type asymmetric registered [ 2.853047] Asymmetric key parser 'x509' registered [ 2.854498] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.857088] io scheduler mq-deadline registered [ 2.859055] io scheduler kyber registered [ 2.861220] io scheduler bfq registered [ 2.863068] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.866279] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.869756] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.872902] ACPI: Power Button [PWRF] [ 2.878082] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.883060] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.894513] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.902061] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.919414] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.949589] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.976776] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.981522] Non-volatile memory driver v1.3 [ 2.982534] Linux agpgart interface v0.103 [ 3.008827] virtio_blk virtio1: [vda] 146936 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.011257] vda: detected capacity change from 0 to 75231232 [ 3.024711] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.027481] vdb: detected capacity change from 0 to 1073741824 [ 3.043635] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.046948] vdc: detected capacity change from 0 to 2621440000 [ 3.062092] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.065294] vdd: detected capacity change from 0 to 2621440000 [ 3.081210] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.084079] vde: detected capacity change from 0 to 4294967296 [ 3.099743] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.102996] vdf: detected capacity change from 0 to 4294967296 [ 3.112296] libphy: Fixed MDIO Bus: probed [ 3.116909] usbcore: registered new interface driver usbserial_generic [ 3.119054] usbserial: USB Serial support registered for generic [ 3.121294] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.125577] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.127686] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.130274] mousedev: PS/2 mouse device common for all mice [ 3.133362] rtc_cmos 00:05: RTC can wake from S4 [ 3.136246] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.140321] rtc_cmos 00:05: registered as rtc0 [ 3.141912] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.144211] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.145595] intel_pstate: CPU model not supported [ 3.151338] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.155903] hid: raw HID events driver (C) Jiri Kosina [ 3.157983] usbcore: registered new interface driver usbhid [ 3.160274] usbhid: USB HID core driver [ 3.161900] drop_monitor: Initializing network drop monitor service [ 3.164490] Initializing XFRM netlink socket [ 3.166446] NET: Registered protocol family 10 [ 3.169540] Segment Routing with IPv6 [ 3.171127] NET: Registered protocol family 17 [ 3.173527] mpls_gso: MPLS GSO support [ 3.179179] RAS: Correctable Errors collector initialized. [ 3.181218] AVX version of gcm_enc/dec engaged. [ 3.182762] AES CTR mode by8 optimization enabled [ 3.250096] sched_clock: Marking stable (3250022568, 0)->(4192617456, -942594888) [ 3.254075] registered taskstats version 1 [ 3.256431] Loading compiled-in X.509 certificates [ 3.258874] zswap: loaded using pool lzo/zbud [ 3.282066] Key type big_key registered [ 3.293387] Key type encrypted registered [ 3.295092] ima: No TPM chip found, activating TPM-bypass! [ 3.297273] ima: Allocated hash algorithm: sha1 [ 3.299080] ima: No architecture policies found [ 3.301090] evm: Initialising EVM extended attributes: [ 3.303145] evm: security.selinux [ 3.304322] evm: security.ima [ 3.305397] evm: security.capability [ 3.306778] evm: HMAC attrs: 0x1 [ 3.309167] rtc_cmos 00:05: setting system clock to 2026-08-27 15:22:34 UTC (1787844154) [ 3.314406] debug: unmapping init [mem 0xffffffff9ba03000-0xffffffff9bbfffff] [ 3.317283] debug: unmapping init [mem 0xffffffff9a782000-0xffffffff9aa58fff] [ 3.328326] Write protecting the kernel read-only data: 28672k [ 3.331478] debug: unmapping init [mem 0xffffffff98e03000-0xffffffff98ffffff] [ 3.334163] debug: unmapping init [mem 0xffffffff99714000-0xffffffff997fffff] [ 3.363727] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.371506] systemd[1]: Detected virtualization kvm. [ 3.373452] systemd[1]: Detected architecture x86-64. [ 3.375169] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.401231] systemd[1]: No hostname configured. [ 3.403045] systemd[1]: Set hostname to . [ 3.405185] random: systemd: uninitialized urandom read (16 bytes read) [ 3.407578] systemd[1]: Initializing machine ID from random generator. [ 3.442239] random: ln: uninitialized urandom read (6 bytes read) [ 3.521631] random: systemd: uninitialized urandom read (16 bytes read) [ 3.524378] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.527649] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.531814] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.066352] device-mapper: uevent: version 1.0.3 [ 4.068427] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 4.557035] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.729916] virtio_net virtio0 ens2: renamed from eth0 [ 4.865983] scsi host0: ata_piix [ 4.886254] scsi host1: ata_piix [ 4.887931] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.890611] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.412301] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.812019] random: crng init done [ 9.813184] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.067688] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.123472] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.361631] SELinux: Disabled at runtime. [ 11.419298] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.431660] systemd[1]: Detected virtualization kvm. [ 11.434733] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.909914] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.913407] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.918351] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.923451] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.927121] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.934208] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.938540] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ 12.035111] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.342858] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.651112] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.668266] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.766446] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.783663] EDAC sbridge: Ver: 1.1.2 [ 14.359791] Key type dns_resolver registered [ 14.653473] NFS: Registering the id_resolver key type [ 14.655536] Key type id_resolver registered [ 14.657211] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg306-server login: [ 39.429817] libcfs: loading out-of-tree module taints kernel. [ 39.449087] Key type ._llcrypt registered [ 39.450749] Key type .llcrypt registered [ 39.490267] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 48.406327] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 49.025058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.033056] alg: No test for adler32 (adler32-zlib) [ 50.056547] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 50.966059] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 52.767256] Key type lgssc registered [ 55.001678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 71.305561] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 109.799854] hrtimer: interrupt took 8786219 ns [ 112.099873] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 125.149172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 125.192258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 126.459533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 126.494312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 126.557927] Lustre: lustre-MDT0000: new disk, initializing [ 126.621145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 126.632601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 131.513430] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 144.127069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 144.220734] Lustre: 6488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 144.268790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 144.273633] Lustre: Skipped 1 previous similar message [ 144.357518] Lustre: lustre-MDT0001: new disk, initializing [ 144.419227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 144.472266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 144.475616] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 149.079459] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.710369] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 162.723443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 163.028652] Lustre: lustre-OST0000: new disk, initializing [ 163.033369] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 163.039768] Lustre: 8426:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 163.109538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 168.659467] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.068110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 173.083766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 173.156925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 182.611384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 182.795497] Lustre: lustre-OST0001: new disk, initializing [ 182.800903] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 182.810667] Lustre: 9499:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 182.887419] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 188.664724] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.088804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 193.120087] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 193.165165] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 198.743638] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 204.021956] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 208.623953] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing check_logdir /tmp/testlogs/ [ 212.946711] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing yml_node [ 216.770428] Lustre: DEBUG MARKER: Client: 2.17.57.89 [ 218.885055] Lustre: DEBUG MARKER: MDS: 2.17.57.89 [ 220.850625] Lustre: DEBUG MARKER: OSS: 2.17.57.89 [ 222.336637] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 11:26:12 EDT 2026 [ 238.622966] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 240.405452] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 256.992249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 256.997812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 257.019706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 260.064455] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 260.065685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 260.109553] Lustre: Skipped 3 previous similar messages [ 265.186035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.197120] Lustre: Skipped 3 previous similar messages [ 270.305112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 270.318798] Lustre: Skipped 3 previous similar messages [ 271.327332] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 271.656380] Lustre: server umount lustre-MDT0000 complete [ 275.464527] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844426 with bad export cookie 2798700031032996361 [ 275.466366] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 275.471634] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 280.548818] LustreError: 8434:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.551788] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.560744] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 280.579184] LustreError: 8434:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 6 previous similar messages [ 280.627457] Lustre: Skipped 3 previous similar messages [ 281.803869] Lustre: server umount lustre-MDT0001 complete [ 286.189583] Lustre: server umount lustre-OST0000 complete [ 290.845091] Lustre: server umount lustre-OST0001 complete [ 298.133594] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 306.789784] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 343.664981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 343.971530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 344.003317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 344.084678] Lustre: lustre-MDT0000: new disk, initializing [ 344.174085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 344.202805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 348.787515] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 358.744355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 358.898953] Lustre: 15053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 358.956568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 358.960521] Lustre: Skipped 1 previous similar message [ 359.043616] Lustre: lustre-MDT0001: new disk, initializing [ 359.152990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 359.193483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 359.207398] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 362.656682] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 372.449390] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 379.597470] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 387.795712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 388.308208] Lustre: lustre-OST0000: new disk, initializing [ 388.314590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 388.321849] Lustre: 17023:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 388.458608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.692842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 389.712945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 389.862319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 395.295639] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 404.815618] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 409.390729] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 409.569799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 414.406730] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 414.642907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 420.837621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 420.839683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.854074] LustreError: Skipped 1 previous similar message [ 420.859569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 420.872190] Lustre: Skipped 1 previous similar message [ 420.890832] Lustre: Skipped 2 previous similar messages [ 422.889932] Lustre: server umount lustre-OST0000 complete [ 431.073032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.076619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 432.385216] Lustre: server umount lustre-MDT0000 complete [ 436.194572] LustreError: 15064:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 436.224953] LustreError: 15064:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 436.972346] LustreError: 15043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844588 with bad export cookie 2798700031032997894 [ 436.980368] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 436.984306] LustreError: 15043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 437.376913] Lustre: server umount lustre-MDT0001 complete [ 442.341187] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 11:29:52 (1787844592) [ 445.384511] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 447.632726] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 11:29:57 (1787844597) [ 456.212548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.518071] LustreError: 19433:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.595066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 460.632609] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 461.797314] LustreError: 19434:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.890483] LustreError: 19433:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 469.030982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.393848] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 480.428807] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 486.467600] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 493.573638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 499.710564] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 507.482566] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 515.552452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 515.561170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.580255] Lustre: Skipped 1 previous similar message [ 515.591756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 515.599114] Lustre: Skipped 2 previous similar messages [ 520.955237] Lustre: server umount lustre-OST0000 complete [ 525.792264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 525.801390] LustreError: Skipped 1 previous similar message [ 530.759762] Lustre: server umount lustre-MDT0000 complete [ 531.941446] LustreError: 21337:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.963501] LustreError: 21337:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 534.757872] LustreError: 19413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844685 with bad export cookie 2798700031032998776 [ 534.767050] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 534.774304] LustreError: 19413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 535.235237] Lustre: server umount lustre-MDT0001 complete [ 547.380567] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 550.216872] Key type lgssc unregistered [ 550.506415] LNet: 22966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 550.516103] LNetError: 22966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 550.538555] LNet: Removed LNI 192.168.203.106@tcp [ 551.641179] Key type .llcrypt unregistered [ 551.647235] Key type ._llcrypt unregistered [ 574.421493] Key type ._llcrypt registered [ 574.423801] Key type .llcrypt registered [ 574.569546] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 575.195889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 575.281993] alg: No test for adler32 (adler32-zlib) [ 576.436536] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 576.693849] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 578.391167] Key type lgssc registered [ 579.465701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 587.928798] loop: module loaded [ 588.047596] loop0: detected capacity change from 0 to 204800000 [ 588.388968] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 588.420704] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 588.827570] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 594.911833] loop0: detected capacity change from 0 to 204800000 [ 595.181554] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 595.198888] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 595.222302] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 600.736776] loop0: detected capacity change from 0 to 204800000 [ 606.467561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 606.523812] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.759461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 607.813113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 607.880849] Lustre: lustre-MDT0000: new disk, initializing [ 607.957396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 607.977324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 613.127931] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 622.116560] loop1: detected capacity change from 0 to 204800000 [ 628.240113] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 628.439374] Lustre: lustre-OST07c6: new disk, initializing [ 628.445370] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 628.450881] Lustre: Skipped 1 previous similar message [ 628.455340] Lustre: 26152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 628.517933] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 635.000853] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 635.517281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 635.533979] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 635.579070] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 640.701752] Lustre: 26438:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 640.709187] Lustre: 26438:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 640.716270] Lustre: 26438:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 640.731414] Lustre: 26438:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 640.741521] Lustre: 26438:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 640.755097] Lustre: 26438:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 655.560986] Lustre: 25138:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 655.570413] Lustre: 25138:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 655.573129] Lustre: 25138:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 655.577715] Lustre: 25138:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 655.587403] Lustre: 25138:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 655.589489] Lustre: 25138:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 660.446339] Lustre: 25139:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 660.456636] Lustre: 25139:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 660.462817] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 660.467805] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 660.474368] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 660.495069] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 660.509948] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 660.525097] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 660.536573] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 660.557706] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 660.563881] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 660.571664] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.823305] Lustre: 25139:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 663.842819] Lustre: 25139:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 663.854279] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 663.864239] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.873744] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 663.884703] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.891881] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 663.910709] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.921693] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 663.935773] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.945443] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 663.957388] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.428424] Lustre: 25138:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 669.453303] Lustre: 25138:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 669.470795] Lustre: 25138:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 669.488264] Lustre: 25138:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.498636] Lustre: 25138:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 669.509299] Lustre: 25138:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.515261] Lustre: 25138:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 669.527304] Lustre: 25138:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.535677] Lustre: 25138:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 669.546580] Lustre: 25138:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.566972] Lustre: 25138:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 669.580198] Lustre: 25138:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 677.658635] Lustre: 25139:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 677.678402] Lustre: 25139:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 677.688968] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 677.702553] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 677.717535] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 677.737440] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 677.748363] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 677.768420] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 677.779743] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 677.790339] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 677.795429] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 677.813746] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 732.651167] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 732.654081] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 732.665162] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 737.769409] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 742.893710] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 743.391135] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 743.568296] Lustre: server umount lustre-OST07c6 complete [ 752.197486] Lustre: server umount lustre-MDT0000 complete [ 768.126673] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 771.075519] Key type lgssc unregistered [ 771.413590] LNet: 28715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.419628] LNetError: 28715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.449355] LNet: Removed LNI 192.168.203.106@tcp [ 772.558660] Key type .llcrypt unregistered [ 772.564969] Key type ._llcrypt unregistered [ 791.071871] Key type ._llcrypt registered [ 791.074636] Key type .llcrypt registered [ 791.179692] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 792.284181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 792.350538] alg: No test for adler32 (adler32-zlib) [ 793.373113] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 793.641061] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 795.359246] Key type lgssc registered [ 796.357538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 803.893594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 803.912961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.528515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 810.425886] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 815.080228] LustreError: 29985:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 815.110214] LustreError: 29985:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 818.831374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.205464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 823.412844] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 833.226988] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 839.648645] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 846.705236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 847.341766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.423962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 854.626097] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 862.386598] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 885.222596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 885.222608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 885.229398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 885.236596] Lustre: Skipped 1 previous similar message [ 888.529778] Lustre: server umount lustre-OST0000 complete [ 895.461782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 895.464846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 895.471622] Lustre: Skipped 1 previous similar message [ 895.492056] Lustre: Skipped 2 previous similar messages [ 899.047480] Lustre: server umount lustre-MDT0000 complete [ 900.581645] LustreError: 29981:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 900.608828] LustreError: 29981:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 902.300590] LustreError: 30720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845053 with bad export cookie 16608542978153761877 [ 902.303680] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 902.308553] LustreError: 30720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 902.633092] Lustre: server umount lustre-MDT0001 complete [ 912.138839] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 914.451656] Key type lgssc unregistered [ 914.742817] LNet: 33728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.750462] LNetError: 33728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.770555] LNet: Removed LNI 192.168.203.106@tcp [ 915.420925] Key type .llcrypt unregistered [ 915.422968] Key type ._llcrypt unregistered [ 935.513814] Key type ._llcrypt registered [ 935.515756] Key type .llcrypt registered [ 935.610125] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 936.705987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 936.742694] alg: No test for adler32 (adler32-zlib) [ 937.789975] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 938.110061] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 939.775301] Key type lgssc registered [ 941.034481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 947.735403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 947.755787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.206794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 952.652561] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 959.457229] LustreError: 34997:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.485022] LustreError: 34997:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 960.169664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.562166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 964.953389] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 972.422808] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 977.534573] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 984.356207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 984.713925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.059588] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 995.011529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 998.277884] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1010.669717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.671247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1010.695699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1010.712869] Lustre: Skipped 1 previous similar message [ 1014.715859] Lustre: server umount lustre-OST0000 complete [ 1020.900250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.912851] Lustre: Skipped 1 previous similar message [ 1020.916569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1020.927841] Lustre: Skipped 1 previous similar message [ 1026.016591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.025273] Lustre: Skipped 1 previous similar message [ 1030.115707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1030.120810] Lustre: Skipped 1 previous similar message [ 1033.183112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1033.367616] Lustre: server umount lustre-MDT0000 complete [ 1035.234322] LustreError: 34996:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.250445] LustreError: 34996:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 1036.452435] LustreError: 35021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845187 with bad export cookie 13696086610356980887 [ 1036.462713] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1036.465956] LustreError: 35021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1036.699282] Lustre: server umount lustre-MDT0001 complete [ 1046.922625] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1050.207276] Key type lgssc unregistered [ 1050.510758] LNet: 38534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.523399] LNetError: 38534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1050.542604] LNet: Removed LNI 192.168.203.106@tcp [ 1051.280891] Key type .llcrypt unregistered [ 1051.285273] Key type ._llcrypt unregistered [ 1083.237680] Key type ._llcrypt registered [ 1083.241140] Key type .llcrypt registered [ 1083.377257] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1084.650766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1084.821362] alg: No test for adler32 (adler32-zlib) [ 1085.873879] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 1086.145397] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1087.824187] Key type lgssc registered [ 1089.166982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1097.312433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1097.328801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.025252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1103.589834] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1109.486707] LustreError: 39952:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1109.519498] LustreError: 39952:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 1112.272567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.597167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1116.958301] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1124.418621] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1130.597835] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1137.730573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1137.976735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1143.788575] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1144.123975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1151.764489] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1162.722539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.738177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1162.744989] Lustre: Skipped 1 previous similar message [ 1164.396817] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 1164.404460] Lustre: Skipped 2 previous similar messages [ 1167.675760] Lustre: server umount lustre-MDT0000 complete [ 1167.842224] LustreError: 39952:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.854258] LustreError: 39952:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 1169.477373] LustreError: 39947:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.314529] LustreError: 39934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845322 with bad export cookie 8689244207259526370 [ 1171.321256] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1171.330185] LustreError: 39934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1172.960251] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1172.960267] LustreError: 42157:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1172.980457] Lustre: Skipped 1 previous similar message [ 1173.013502] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1176.993168] LustreError: 39948:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.994129] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1177.008498] LustreError: 39948:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 1177.020321] Lustre: Skipped 1 previous similar message [ 1177.647219] Lustre: server umount lustre-MDT0001 complete [ 1198.680025] Lustre: server umount lustre-OST0000 complete [ 1215.096069] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1218.469017] Key type lgssc unregistered [ 1218.830805] LNet: 43816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1218.837692] LNetError: 43816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.845685] LNet: Removed LNI 192.168.203.106@tcp [ 1219.750723] Key type .llcrypt unregistered [ 1219.755846] Key type ._llcrypt unregistered [ 1239.368485] Key type ._llcrypt registered [ 1239.370456] Key type .llcrypt registered [ 1239.454475] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1240.174034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1240.189203] alg: No test for adler32 (adler32-zlib) [ 1241.193929] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 1241.393226] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1243.031191] Key type lgssc registered [ 1243.947229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1250.326965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1250.354541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.743075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.456766] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1261.091784] LustreError: 45212:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.123075] LustreError: 45212:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 1263.361910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1263.765857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1268.090974] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1276.205915] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1283.730172] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1290.465840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1290.786568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.929469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1296.351459] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1302.551615] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1313.249386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1313.262062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.270346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1313.273149] Lustre: Skipped 1 previous similar message [ 1313.308759] Lustre: Skipped 1 previous similar message [ 1314.936803] Lustre: server umount lustre-OST0000 complete [ 1333.731925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.758896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1333.768866] Lustre: Skipped 1 previous similar message [ 1334.817924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1334.823019] Lustre: Skipped 1 previous similar message [ 1339.255186] Lustre: server umount lustre-MDT0000 complete [ 1339.878846] LustreError: 45213:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.906211] LustreError: 45213:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 1342.841345] LustreError: 45192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845494 with bad export cookie 17967757335934913919 [ 1342.846423] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1342.848580] LustreError: 45192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1357.279129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1357.479755] Lustre: server umount lustre-MDT0001 complete [ 1370.540620] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1373.192362] Key type lgssc unregistered [ 1373.525657] LNet: 48928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.536353] LNetError: 48928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1373.544877] LNet: Removed LNI 192.168.203.106@tcp [ 1374.617379] Key type .llcrypt unregistered [ 1374.623029] Key type ._llcrypt unregistered [ 1398.301990] Key type ._llcrypt registered [ 1398.307506] Key type .llcrypt registered [ 1398.387519] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1399.312609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1399.397208] alg: No test for adler32 (adler32-zlib) [ 1400.541577] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 1400.738720] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1402.447349] Key type lgssc registered [ 1403.640157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1411.912136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1411.926906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1413.446964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.071389] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1422.825481] LustreError: 50345:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.839773] LustreError: 50345:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 1427.550567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1428.034300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1433.027031] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1441.694196] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1447.339135] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1454.376474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1454.811082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1457.858236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1461.201730] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1470.490558] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1479.599702] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1484.469967] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1497.442577] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2711h47m26s [ 1499.323026] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1503.341891] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 1504.244605] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1504.247472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1504.285674] Lustre: Skipped 2 previous similar messages [ 1504.310136] Lustre: Skipped 2 previous similar messages [ 1505.379748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1505.385950] Lustre: Skipped 3 previous similar messages [ 1507.512640] LustreError: 53245:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000830ecbc3) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1509.643831] Lustre: server umount lustre-MDT0000 complete [ 1510.462517] LustreError: 51330:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1511.913684] LustreError: 50345:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1511.940615] LustreError: 50345:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 1513.313524] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1516.609795] LustreError: 51330:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1520.790410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.947218] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1521.268626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.356988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1522.282590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1525.921468] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1529.540591] Lustre: DEBUG MARKER: Wait for df (36721) ... [ 1531.553472] Lustre: DEBUG MARKER: done [ 1538.025692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1538.045702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1538.055619] Lustre: Skipped 1 previous similar message [ 1538.072524] Lustre: Skipped 1 previous similar message [ 1542.942612] Lustre: server umount lustre-OST0000 complete [ 1548.257694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1548.258960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1548.259418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1548.270155] Lustre: Skipped 1 previous similar message [ 1559.527698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1559.533414] Lustre: Skipped 6 previous similar messages [ 1561.057096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1561.322663] Lustre: server umount lustre-MDT0000 complete [ 1564.594587] LustreError: 52643:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.624397] LustreError: 52643:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 4 previous similar messages [ 1565.613318] LustreError: 52976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845716 with bad export cookie 5372612958170826654 [ 1565.620266] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1565.633232] LustreError: 52976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1566.141318] Lustre: server umount lustre-MDT0001 complete [ 1577.839504] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1580.564421] Key type lgssc unregistered [ 1580.876711] LNet: 55277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1580.882644] LNetError: 55277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.903387] LNet: Removed LNI 192.168.203.106@tcp [ 1581.750356] Key type .llcrypt unregistered [ 1581.751917] Key type ._llcrypt unregistered [ 1623.865760] Key type ._llcrypt registered [ 1623.868446] Key type .llcrypt registered [ 1623.995921] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1624.947326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1625.035099] alg: No test for adler32 (adler32-zlib) [ 1626.085553] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 1626.337747] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1628.063391] Key type lgssc registered [ 1629.373605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1637.422210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1637.436924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.693568] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1638.732161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1638.877884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1643.384461] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1651.783419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1651.900086] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1651.937685] Lustre: 57489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1652.275781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1656.995768] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1664.309572] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1669.538807] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1675.925220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1676.153438] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1676.345561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.050454] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1685.703227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1689.642541] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1697.079475] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2711h50m47s [ 1698.736812] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1706.406170] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1709.677832] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1709.727194] LustreError: 57496:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1730.170341] Lustre: lustre-MDT0000: Client c9f7ff32-705e-441f-9313-ea162a1efb56 (at 192.168.203.6@tcp) reconnecting [ 1749.751708] LustreError: 57496:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1752.026981] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1754.365539] Lustre: DEBUG MARKER: done [ 1762.104474] Lustre: server umount lustre-OST0000 complete [ 1767.906362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.911840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1767.919140] Lustre: Skipped 1 previous similar message [ 1767.933408] Lustre: Skipped 1 previous similar message [ 1768.993603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1768.998086] Lustre: Skipped 1 previous similar message [ 1775.079918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1775.089941] Lustre: Skipped 1 previous similar message [ 1775.135334] Lustre: server umount lustre-MDT0000 complete [ 1778.153200] Lustre: 56811:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845913/real 1787845913] req@000000006aa67733 x1874692173519232/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787845929 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1778.191176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1778.828502] LustreError: 58231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845930 with bad export cookie 4065290087520559207 [ 1778.834552] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1778.847225] LustreError: 58231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1779.248958] Lustre: server umount lustre-MDT0001 complete [ 1789.637351] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1792.188412] Key type lgssc unregistered [ 1792.485685] LNet: 61820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1792.501494] LNetError: 61820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1792.543820] LNet: Removed LNI 192.168.203.106@tcp [ 1793.365772] Key type .llcrypt unregistered [ 1793.369647] Key type ._llcrypt unregistered [ 1837.255187] Key type ._llcrypt registered [ 1837.259579] Key type .llcrypt registered [ 1837.389969] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1838.633342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1838.753328] alg: No test for adler32 (adler32-zlib) [ 1839.926532] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 1840.113870] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1841.743362] Key type lgssc registered [ 1842.797776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1846.270713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1846.324924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1847.563126] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1847.575865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1847.682695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.672609] Lustre: Failing over lustre-MDT0000 [ 1848.911585] Lustre: server umount lustre-MDT0000 complete [ 1856.268430] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:53:26 (1787846006) [ 1864.115616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.588044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.372218] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1877.962037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.082417] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1878.126774] Lustre: 64620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1878.229843] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1878.424694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1882.674360] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1889.791598] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1895.863872] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1902.392741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1902.579597] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1902.843929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1909.179536] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1909.951801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1917.314656] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1922.835318] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1930.721186] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.728059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1930.741422] Lustre: Skipped 2 previous similar messages [ 1930.762853] Lustre: Skipped 2 previous similar messages [ 1935.854311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1935.862731] Lustre: Skipped 2 previous similar messages [ 1936.504340] Lustre: server umount lustre-MDT0000 complete [ 1941.086555] LustreError: 64611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846092 with bad export cookie 9935407633437943014 [ 1941.088507] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1941.102562] LustreError: 64611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1941.415412] Lustre: server umount lustre-MDT0001 complete [ 1945.941629] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1951.182967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1959.961846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.463777] Lustre: 63542:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846097/real 1787846097] req@0000000026deffb6 x1874692397899904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846113 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1962.493674] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.625686] LustreError: 63539:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f1919a89 x1874692397900672/t0(0) o250->MGC192.168.203.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1966.945358] LustreError: 66470:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.968842] LustreError: 66470:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 4 previous similar messages [ 1967.049239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1967.166891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1968.031152] Lustre: 63543:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846102/real 1787846102] req@00000000656c63f3 x1874692397900032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846118 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1971.007979] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1971.182320] LustreError: 66470:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1971.185543] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1971.198165] LustreError: 66470:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 1972.129121] Lustre: 63541:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846107/real 1787846107] req@000000002679ad06 x1874692397900288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846123 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1972.260022] LustreError: 66471:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1972.288014] LustreError: 66471:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 1977.333724] LustreError: 66470:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.358795] LustreError: 66470:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 1978.335799] Lustre: 63540:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846112/real 1787846112] req@00000000806e7436 x1874692397900416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846128 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1980.656402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.052427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1986.272984] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1986.559468] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1994.374304] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2000.306267] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2011.628110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2011.639141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.665903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2013.809680] Lustre: server umount lustre-MDT0000 complete [ 2017.250612] LustreError: 66470:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.263056] LustreError: 66470:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2017.846279] LustreError: 66486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846169 with bad export cookie 9935407633437945289 [ 2017.855606] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2017.859274] LustreError: 66486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2018.169822] Lustre: server umount lustre-MDT0001 complete [ 2024.053349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2032.478794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.751567] Lustre: 63542:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846173/real 1787846173] req@000000003ea2976b x1874692397928320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846189 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2038.792574] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.820409] Lustre: Skipped 2 previous similar messages [ 2043.266950] LustreError: 66470:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.343442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.407534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2048.312679] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2050.464523] Lustre: 63541:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846184/real 1787846184] req@000000001b40be73 x1874692397928832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2050.517753] Lustre: 63541:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2057.310758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.837163] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2062.997246] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2066.924670] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2071.674986] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2077.864195] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2087.396759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2087.417904] Lustre: Skipped 1 previous similar message [ 2087.418766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2087.435307] Lustre: Skipped 5 previous similar messages [ 2091.708577] Lustre: server umount lustre-MDT0000 complete [ 2092.513065] LustreError: 71073:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2092.538938] LustreError: 71073:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 7 previous similar messages [ 2095.727454] LustreError: 64610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846246 with bad export cookie 9935407633437946668 [ 2095.744227] LustreError: 64610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2095.745725] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2096.060761] Lustre: server umount lustre-MDT0001 complete [ 2103.364091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2107.210402] Lustre: DEBUG MARKER: files compared the same [ 2119.748766] Lustre: server umount lustre-OST0000 complete [ 2137.217430] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2140.278837] Key type lgssc unregistered [ 2140.670585] LNet: 74355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2140.681435] LNetError: 74355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2140.710061] LNet: Removed LNI 192.168.203.106@tcp [ 2141.796190] Key type .llcrypt unregistered [ 2141.799859] Key type ._llcrypt unregistered [ 2169.939401] Key type ._llcrypt registered [ 2169.942362] Key type .llcrypt registered [ 2170.040972] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2170.973740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2171.089186] alg: No test for adler32 (adler32-zlib) [ 2172.227950] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2172.499297] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2174.207277] Key type lgssc registered [ 2175.397495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2183.342931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2183.371973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2184.827863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.491467] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug +malloc all [ 2194.400430] LustreError: 75772:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.419045] LustreError: 75772:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2198.135444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.378395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2203.189960] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug +malloc all [ 2211.220215] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2217.441508] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2224.354493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2224.585782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.740954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2231.833767] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug +malloc all [ 2239.755200] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2253.282637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2253.294053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2253.308455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2253.321542] Lustre: Skipped 1 previous similar message [ 2253.344616] Lustre: Skipped 1 previous similar message [ 2258.414275] LustreError: 77594:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.580996] Lustre: server umount lustre-OST0000 complete [ 2263.522618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.535640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2263.537283] Lustre: Skipped 1 previous similar message [ 2263.556790] Lustre: Skipped 1 previous similar message [ 2268.658573] LustreError: 75769:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.682058] LustreError: 75769:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2268.960953] Lustre: server umount lustre-MDT0000 complete [ 2272.633362] LustreError: 75752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846423 with bad export cookie 5096096757866712499 [ 2272.636344] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2272.647688] LustreError: 75752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2273.122714] Lustre: server umount lustre-MDT0001 complete [ 2284.960037] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2288.116623] Key type lgssc unregistered [ 2288.407044] LNet: 79331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2288.418603] LNetError: 79331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2289.459608] LNet: Removed LNI 192.168.203.106@tcp [ 2290.466800] Key type .llcrypt unregistered [ 2290.469167] Key type ._llcrypt unregistered [ 2315.433328] Key type ._llcrypt registered [ 2315.436702] Key type .llcrypt registered [ 2315.564356] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2317.119314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2317.135765] alg: No test for adler32 (adler32-zlib) [ 2318.215034] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2318.621469] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2320.380499] Key type lgssc registered [ 2321.709459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2329.849220] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2329.888263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2345.504270] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2350.623827] LustreError: 80736:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 2377.311192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.948920] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2391.406954] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2400.020046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.227520] Lustre: *** cfs_fail_loc=706, val=0*** [ 2400.851958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2400.932448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2405.861154] LustreError: 80762:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.887962] LustreError: 80762:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2405.994588] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2410.975904] LustreError: 80761:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.011463] LustreError: 80762:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.045535] LustreError: 80762:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 2414.977465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.204902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2418.626277] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2426.189104] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2434.487770] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2440.672993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2440.679154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2440.720851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2440.756258] Lustre: Skipped 1 previous similar message [ 2443.841322] Lustre: server umount lustre-OST0000 complete [ 2450.914357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.940208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.940326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2450.955898] Lustre: Skipped 2 previous similar messages [ 2455.005519] Lustre: server umount lustre-MDT0000 complete [ 2456.033170] LustreError: 81827:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.073072] LustreError: 81827:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 2458.644434] LustreError: 80743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846609 with bad export cookie 14287164373105853149 [ 2458.654215] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2458.656690] LustreError: 80743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2458.991060] Lustre: server umount lustre-MDT0001 complete [ 2470.226110] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2473.313419] Key type lgssc unregistered [ 2473.699224] LNet: 84356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.704541] LNetError: 84356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2473.725943] LNet: Removed LNI 192.168.203.106@tcp [ 2474.721392] Key type .llcrypt unregistered [ 2474.724104] Key type ._llcrypt unregistered [ 2501.930410] Key type ._llcrypt registered [ 2501.932416] Key type .llcrypt registered [ 2502.046586] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2503.189175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2503.196837] alg: No test for adler32 (adler32-zlib) [ 2504.460507] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2504.691817] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2506.345307] Key type lgssc registered [ 2507.636678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2517.164651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2517.190415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.283339] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2529.569958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.270087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2534.734177] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2535.395470] LustreError: 86462:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.412708] LustreError: 86462:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 2540.512301] LustreError: 86466:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.546145] LustreError: 86466:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 2543.250035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2543.814529] LustreError: 86462:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.833413] LustreError: 86462:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2543.845441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.219116] LustreError: 86465:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.252227] LustreError: 86465:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2551.653435] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2554.356550] LustreError: 86466:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.391484] LustreError: 86466:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2561.957810] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2561.973566] Lustre: MGS: Client a3adbaca-629a-47f1-a0b7-797ce698a045 (at 0@lo) reconnecting [ 2561.988218] LustreError: 86476:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2562.381089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.496081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2567.036904] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2583.014385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2583.015630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2583.018270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2583.044316] Lustre: Skipped 1 previous similar message [ 2588.153545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2588.158626] Lustre: Skipped 1 previous similar message [ 2588.877357] Lustre: server umount lustre-OST0000 complete [ 2593.251093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.254592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2593.255113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2593.255118] Lustre: Skipped 1 previous similar message [ 2593.281665] Lustre: Skipped 1 previous similar message [ 2598.375408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2598.384533] Lustre: Skipped 2 previous similar messages [ 2598.975520] Lustre: server umount lustre-MDT0000 complete [ 2603.760928] Lustre: server umount lustre-MDT0001 complete [ 2608.191940] Lustre: server umount MGS complete [ 2620.625393] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:06:10 (1787846770) [ 2650.977508] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 2661.760537] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2699.896709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.568193] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2715.988117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.335194] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2716.436448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2716.596448] Lustre: lustre-MDT0001: new disk, initializing [ 2716.694539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2716.737100] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2716.749559] Lustre: 93652:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2716.767850] LustreError: 93652:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2721.848574] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2733.491583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2733.727363] Lustre: lustre-OST0000: new disk, initializing [ 2733.732231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2733.737231] Lustre: 94420:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2733.780966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.458191] Lustre: 94715:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000de751928 x1874693094155392/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787846910 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2743.881318] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2757.924851] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2763.745480] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2764.635075] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2764.665705] Lustre: MGS: Client 4e1c09ec-6447-495a-922a-638294e8343e (at 0@lo) reconnecting [ 2764.729201] Lustre: 95466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2764.746730] LustreError: 93638:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2764.790057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2764.914837] Lustre: lustre-MDT0000: new disk, initializing [ 2765.036374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2765.115618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2766.181607] Lustre: 95490:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000009ae1153f x1874693094163456/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787846933 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2769.590564] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2771.295527] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2775.338734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2775.363393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2775.522043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2801.124302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2801.129529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2801.144549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2804.708371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2806.987326] Lustre: server umount lustre-OST0000 complete [ 2814.950847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.963656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2814.975195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2814.990024] Lustre: Skipped 3 previous similar messages [ 2816.841928] Lustre: server umount lustre-MDT0000 complete [ 2820.066382] LustreError: 93623:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2820.084921] LustreError: 93623:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 7 previous similar messages [ 2826.861476] Lustre: server umount lustre-MDT0001 complete [ 2830.501295] Lustre: server umount MGS complete [ 2842.604085] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 12:09:52 (1787846992) [ 2860.625964] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2864.140130] Key type lgssc unregistered [ 2864.483664] LNet: 97999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2864.491637] LNetError: 97999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2864.543114] LNet: Removed LNI 192.168.203.106@tcp [ 2865.446387] Key type .llcrypt unregistered [ 2865.450973] Key type ._llcrypt unregistered [ 2889.133795] Key type ._llcrypt registered [ 2889.135941] Key type .llcrypt registered [ 2889.326506] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2891.283755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2891.361582] alg: No test for adler32 (adler32-zlib) [ 2892.492561] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2892.818066] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2894.519172] Key type lgssc registered [ 2895.833343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2905.719248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2905.733629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.350112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2912.790363] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2916.834271] LustreError: 99224:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.858733] LustreError: 99224:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2921.954472] LustreError: 99225:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.798140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2924.160624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2930.750279] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2939.829753] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2946.862721] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2954.203518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2954.599292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2961.366570] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2969.059846] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2977.141867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2987.504128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2987.504735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2987.537529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2992.430700] Lustre: server umount lustre-OST0000 complete [ 2996.194576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2996.200234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2996.213754] Lustre: Skipped 1 previous similar message [ 2996.217352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2996.224558] Lustre: Skipped 1 previous similar message [ 2997.728446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.729221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2997.744051] Lustre: Skipped 1 previous similar message [ 3002.305885] Lustre: server umount lustre-MDT0000 complete [ 3002.851596] LustreError: 101316:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.873051] LustreError: 101316:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 3006.289112] LustreError: 99206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847157 with bad export cookie 12393689518020503766 [ 3006.304373] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3006.648558] Lustre: server umount lustre-MDT0001 complete [ 3020.199641] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3023.606088] Key type lgssc unregistered [ 3023.914405] LNet: 102781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3023.927506] LNetError: 102781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3023.944418] LNet: Removed LNI 192.168.203.106@tcp [ 3024.841121] Key type .llcrypt unregistered [ 3024.847630] Key type ._llcrypt unregistered [ 3042.841912] Key type ._llcrypt registered [ 3042.847312] Key type .llcrypt registered [ 3042.954684] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3043.921517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3044.205043] alg: No test for adler32 (adler32-zlib) [ 3045.312191] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 3045.655728] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3047.359292] Key type lgssc registered [ 3048.328399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3054.942970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3054.956566] LustreError: 103853:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3055.018279] LustreError: 103866:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3055.025677] LustreError: 103853:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4946 [ 3055.034426] LustreError: 103853:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3055.047484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.048621] LustreError: 103853:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3056.544978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.653518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3065.824321] LustreError: 103904:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.854992] LustreError: 103904:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 3066.116737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3070.737501] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3076.484568] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing lsmod [ 3079.558410] LustreError: 105072:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3079.589425] LustreError: 105073:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3079.597186] LustreError: 105072:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4976 [ 3079.614641] LustreError: 105072:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3079.620970] LustreError: 105072:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3079.650659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3080.179330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.214775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3086.315780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3086.317581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3086.333036] Lustre: Skipped 2 previous similar messages [ 3086.352394] Lustre: Skipped 2 previous similar messages [ 3089.617895] Lustre: server umount lustre-MDT0000 complete [ 3091.431749] LustreError: 103899:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3091.451841] LustreError: 103899:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 3092.449309] LustreError: 103905:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3092.472443] LustreError: 103905:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 3093.575083] LustreError: 103884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847244 with bad export cookie 2417119715598368718 [ 3093.583918] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3093.596936] LustreError: 103884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3093.617970] Lustre: 103896:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.106@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3094.559291] LustreError: 104335:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008e2d29e0 x1874693661414016/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3094.888099] Lustre: server umount lustre-MDT0001 complete [ 3109.503194] Lustre: server umount lustre-OST0000 complete [ 3118.605168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.151907] LustreError: 106137:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.253223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3124.122248] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3124.714707] LustreError: 106138:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.172025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3133.676668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3138.846351] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3147.391662] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3153.714492] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3160.998296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3161.398657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3165.490148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3168.623273] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3178.223844] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3186.144491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3186.164347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3186.192448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3189.216520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3189.216927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3189.224131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.765937] Lustre: server umount lustre-OST0000 complete [ 3199.471755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.473381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3199.492556] Lustre: Skipped 1 previous similar message [ 3199.505715] Lustre: Skipped 2 previous similar messages [ 3202.709952] Lustre: server umount lustre-MDT0000 complete [ 3204.578322] LustreError: 106137:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3204.609591] LustreError: 106137:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 3206.778621] LustreError: 106117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847357 with bad export cookie 2417119715598369635 [ 3206.787936] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3206.793599] LustreError: 106117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3207.160911] Lustre: server umount lustre-MDT0001 complete [ 3221.032955] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3224.611388] Key type lgssc unregistered [ 3225.029254] LNet: 109674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3225.062422] LNetError: 109674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3225.086966] LNet: Removed LNI 192.168.203.106@tcp [ 3226.099397] Key type .llcrypt unregistered [ 3226.102476] Key type ._llcrypt unregistered [ 3252.046697] Key type ._llcrypt registered [ 3252.052036] Key type .llcrypt registered [ 3252.206828] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3253.006751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3253.098498] alg: No test for adler32 (adler32-zlib) [ 3254.177861] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 3254.471325] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3256.185878] Key type lgssc registered [ 3257.266842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3267.938264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3267.964849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.647608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3275.215247] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3278.833431] LustreError: 111094:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3278.895405] LustreError: 111094:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 3283.937518] LustreError: 111093:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3285.064350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.492562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3291.121318] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3300.424948] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3306.470299] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3314.199460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3314.732172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3321.655766] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3329.524417] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3339.318239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3344.531427] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3348.165196] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3354.594972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.597870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3354.604203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3354.628815] Lustre: Skipped 1 previous similar message [ 3359.320798] Lustre: server umount lustre-OST0000 complete [ 3364.834712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3364.851161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3364.858706] Lustre: Skipped 1 previous similar message [ 3364.868512] Lustre: Skipped 2 previous similar messages [ 3369.958932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3370.571467] Lustre: server umount lustre-MDT0000 complete [ 3374.048978] LustreError: 112048:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3374.060200] LustreError: 112048:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 3374.768398] LustreError: 111075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847525 with bad export cookie 16970874387111849666 [ 3374.770899] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3374.779273] LustreError: 111075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3375.212086] Lustre: server umount lustre-MDT0001 complete [ 3387.949365] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3390.923538] Key type lgssc unregistered [ 3391.232950] LNet: 114758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3391.239854] LNetError: 114758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3391.266721] LNet: Removed LNI 192.168.203.106@tcp [ 3392.234611] Key type .llcrypt unregistered [ 3392.236991] Key type ._llcrypt unregistered [ 3411.077562] Key type ._llcrypt registered [ 3411.079597] Key type .llcrypt registered [ 3411.162847] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3412.356658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3412.370256] alg: No test for adler32 (adler32-zlib) [ 3413.407312] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 3413.658307] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3415.335175] Key type lgssc registered [ 3416.581397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3424.198807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3424.212491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3425.644471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.282459] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3434.993111] LustreError: 115980:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3435.014473] LustreError: 115980:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 3440.103992] LustreError: 115979:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3440.383531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3440.748609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3445.242368] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3453.623990] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3460.642965] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3468.249573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3468.648891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3470.713831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3474.965170] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3484.209888] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3495.395609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3495.395944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.409802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3495.425296] Lustre: Skipped 1 previous similar message [ 3499.552345] Lustre: server umount lustre-OST0000 complete [ 3505.632583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3505.646514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3505.658066] Lustre: Skipped 1 previous similar message [ 3505.671739] Lustre: Skipped 2 previous similar messages [ 3510.752552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3514.857584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3514.866379] Lustre: Skipped 2 previous similar messages [ 3517.408667] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3517.604316] Lustre: server umount lustre-MDT0000 complete [ 3519.980389] LustreError: 115980:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3520.003681] LustreError: 115980:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 3522.120323] LustreError: 115961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847673 with bad export cookie 13469666576868597125 [ 3522.129371] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3522.132875] LustreError: 115961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3522.616751] Lustre: server umount lustre-MDT0001 complete [ 3533.109421] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3535.731570] Key type lgssc unregistered [ 3536.149716] LNet: 119519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3536.155962] LNetError: 119519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3536.167573] LNet: Removed LNI 192.168.203.106@tcp [ 3537.109058] Key type .llcrypt unregistered [ 3537.116149] Key type ._llcrypt unregistered [ 3561.900734] Key type ._llcrypt registered [ 3561.903530] Key type .llcrypt registered [ 3562.011058] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3563.032726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3563.235783] alg: No test for adler32 (adler32-zlib) [ 3564.360855] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 3564.694411] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3566.367239] Key type lgssc registered [ 3567.544427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3576.115640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3576.145228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3577.654942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3582.491941] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3587.047162] LustreError: 120940:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3587.092808] LustreError: 120940:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 3591.904415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.160155] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3592.308229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3597.488409] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3606.854587] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3613.416216] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3621.033643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3621.511830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3622.545682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3627.714909] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3638.059468] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3646.472250] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3650.240857] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3650.261258] Lustre: Skipped 2 previous similar messages [ 3652.702237] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3652.749805] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3654.941968] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3654.948914] Lustre: Skipped 2 previous similar messages [ 3658.114547] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3658.120268] Lustre: Skipped 1 previous similar message [ 3667.695516] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.6@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3667.713700] Lustre: Skipped 2 previous similar messages [ 3677.274663] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.6@tcp 192.168.0.[2,10]@tcp [ 3680.338781] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.6/24@tcp in log lustre-MDT0000 [ 3680.343988] Lustre: Skipped 2 previous similar messages [ 3683.390370] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.6/24@tcp [ 3683.398726] Lustre: Skipped 1 previous similar message [ 3693.538139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3693.543975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3693.563614] Lustre: Skipped 1 previous similar message [ 3698.166826] Lustre: server umount lustre-OST0000 complete [ 3703.778482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3703.781431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3703.800456] Lustre: Skipped 1 previous similar message [ 3703.815900] Lustre: Skipped 2 previous similar messages [ 3708.460724] Lustre: server umount lustre-MDT0000 complete [ 3708.898899] LustreError: 121938:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3708.921904] LustreError: 121938:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 3711.950722] LustreError: 120922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847863 with bad export cookie 4952319187639993836 [ 3711.953412] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3711.965119] LustreError: 120922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3712.270338] Lustre: server umount lustre-MDT0001 complete [ 3722.834315] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3725.452863] Key type lgssc unregistered [ 3725.773675] LNet: 125885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3725.778232] LNetError: 125885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3725.797269] LNet: Removed LNI 192.168.203.106@tcp [ 3726.632868] Key type .llcrypt unregistered [ 3726.647758] Key type ._llcrypt unregistered [ 3756.862853] Key type ._llcrypt registered [ 3756.869272] Key type .llcrypt registered [ 3757.105894] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3758.482699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3758.749834] alg: No test for adler32 (adler32-zlib) [ 3760.018600] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 3760.234033] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3761.855124] Key type lgssc registered [ 3763.336445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3771.288298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3771.307632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3772.704671] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3772.732630] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.6/24@tcp [ 3772.772126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3776.329981] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3782.120368] LustreError: 127446:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3782.130567] LustreError: 127446:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 3783.682445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.922067] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3783.944880] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.6/24@tcp [ 3784.008646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3788.661344] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3795.477135] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3801.464847] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3807.829566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3808.151422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3813.585353] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3817.409798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3821.757692] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3832.810324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.818114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3832.833697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3837.925707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3837.929564] Lustre: Skipped 2 previous similar messages [ 3838.318099] Lustre: server umount lustre-OST0000 complete [ 3843.042552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.047194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.074547] Lustre: Skipped 2 previous similar messages [ 3848.162731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3848.173927] Lustre: Skipped 2 previous similar messages [ 3853.293385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3853.304876] Lustre: Skipped 1 previous similar message [ 3856.351472] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3856.634498] Lustre: server umount lustre-MDT0000 complete [ 3858.412054] LustreError: 127443:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3858.437601] LustreError: 127443:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 3860.656725] LustreError: 127428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848011 with bad export cookie 474668705333607842 [ 3860.659461] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3860.674092] LustreError: 127428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3860.967768] Lustre: server umount lustre-MDT0001 complete [ 3871.027406] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3873.836985] Key type lgssc unregistered [ 3874.111052] LNet: 131040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3874.119560] LNetError: 131040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3874.146907] LNet: Removed LNI 192.168.203.106@tcp [ 3874.915472] Key type .llcrypt unregistered [ 3874.921335] Key type ._llcrypt unregistered [ 3898.418529] Key type ._llcrypt registered [ 3898.420644] Key type .llcrypt registered [ 3898.594840] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 3918.107632] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3919.198569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3919.221482] alg: No test for adler32 (adler32-zlib) [ 3920.304808] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 3920.516775] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3922.167139] Key type lgssc registered [ 3923.328771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3958.313193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3958.356268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3959.520480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3959.547251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3959.615239] Lustre: lustre-MDT0000: new disk, initializing [ 3959.690929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3959.708888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3964.626926] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3973.974614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3974.114138] Lustre: 134444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3974.182046] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3974.187095] Lustre: Skipped 1 previous similar message [ 3974.295121] Lustre: lustre-MDT0001: new disk, initializing [ 3974.412182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3974.458192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3974.473047] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3978.907691] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3987.152414] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3992.169697] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3999.158462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3999.495138] Lustre: lustre-OST0000: new disk, initializing [ 3999.505866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3999.523373] Lustre: 136410:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3999.627636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4005.655579] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4007.493658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4007.517698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4007.658717] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4015.215067] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4019.960476] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4020.163343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4024.488269] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4024.739752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4028.384335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4028.400177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4028.402245] LustreError: Skipped 1 previous similar message [ 4028.403089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4028.433488] Lustre: Skipped 1 previous similar message [ 4032.843571] Lustre: server umount lustre-OST0000 complete [ 4038.625661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4038.641746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4038.653559] Lustre: Skipped 1 previous similar message [ 4038.677851] Lustre: Skipped 2 previous similar messages [ 4043.133766] Lustre: server umount lustre-MDT0000 complete [ 4046.976902] LustreError: 134434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848198 with bad export cookie 16919432301095541361 [ 4046.988102] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4046.990539] LustreError: 134434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4047.350731] Lustre: server umount lustre-MDT0001 complete [ 4055.610639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4056.189209] LustreError: 138628:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4056.206819] LustreError: 138628:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 4056.238272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4060.864972] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4061.676220] LustreError: 138629:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4066.790794] LustreError: 138628:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.690841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4069.069363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4072.821094] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4078.672264] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4083.976682] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4090.133908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4090.520931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4096.916083] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4104.155596] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4109.002352] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4109.224762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4114.450148] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4114.722517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4126.649433] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4126.965894] Lustre: lustre-OST0001: new disk, initializing [ 4126.970604] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4126.975860] Lustre: 142003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4127.057859] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4135.084354] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4135.113434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4135.135660] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4135.317508] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4143.413453] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4143.704804] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4148.433722] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4148.637637] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4154.326722] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8909888fb800.ost_server_uuid 50 [ 4156.297789] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8909888fb800.ost_server_uuid in IDLE state after 0 sec [ 4166.117813] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4166.126497] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4166.140529] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4166.151454] Lustre: Skipped 1 previous similar message [ 4171.093725] Lustre: server umount lustre-OST0001 complete [ 4171.265353] LustreError: 141071:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4171.284936] LustreError: 141071:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 4176.358794] LustreError: 140454:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4176.364033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4176.370320] LustreError: 140454:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 4176.399686] Lustre: Skipped 1 previous similar message [ 4176.404911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4176.408511] Lustre: Skipped 1 previous similar message [ 4181.316335] Lustre: server umount lustre-OST0000 complete [ 4186.592670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4186.602528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4186.621222] Lustre: Skipped 2 previous similar messages [ 4186.640623] Lustre: Skipped 1 previous similar message [ 4190.705785] Lustre: server umount lustre-MDT0000 complete [ 4191.713802] LustreError: 140057:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4191.736339] LustreError: 140057:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 4194.163607] LustreError: 138653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848345 with bad export cookie 16919432301095542243 [ 4194.181602] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4194.483642] Lustre: server umount lustre-MDT0001 complete [ 4213.550972] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4216.143671] Key type lgssc unregistered [ 4216.448461] LNet: 145008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4216.460713] LNetError: 145008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4216.477881] LNet: Removed LNI 192.168.203.106@tcp [ 4217.263225] Key type .llcrypt unregistered [ 4217.266630] Key type ._llcrypt unregistered [ 4257.791959] Key type ._llcrypt registered [ 4257.796162] Key type .llcrypt registered [ 4257.915469] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 4271.616913] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4272.276823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4272.371827] alg: No test for adler32 (adler32-zlib) [ 4273.417830] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 4273.674951] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 4275.335339] Key type lgssc registered [ 4276.484699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4310.011020] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4310.082226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.366668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4311.398510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4311.501917] Lustre: lustre-MDT0000: new disk, initializing [ 4311.618392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4311.647926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4316.285079] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4325.842283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4325.987137] Lustre: 149165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4326.090963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4326.102489] Lustre: Skipped 1 previous similar message [ 4326.176456] Lustre: lustre-MDT0001: new disk, initializing [ 4326.284902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4326.351705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4326.376656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4331.332920] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4340.321972] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4346.064602] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4352.159929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4352.444805] Lustre: lustre-OST0000: new disk, initializing [ 4352.450278] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4352.460452] Lustre: 151133:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4352.567561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4358.398639] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4359.246487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4359.261738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4359.357853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4368.882664] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4373.926630] Lustre: Failing over lustre-OST0000 [ 4374.159713] Lustre: server umount lustre-OST0000 complete [ 4391.391243] Lustre: 147389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787848526/real 1787848526] req@000000001edf3cec x1874694949078144/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787848542 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4391.392787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4391.407454] Lustre: 147389:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4391.458526] Lustre: Skipped 1 previous similar message [ 4391.858479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4392.038582] Lustre: lustre-OST0000: Not available for connect from 192.168.203.6@tcp (not set up) [ 4392.146337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.160542] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4393.111109] Lustre: 147387:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787848527/real 1787848527] req@00000000f4c9f995 x1874694949078400/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787848543 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4393.132573] Lustre: 147387:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4396.001932] Lustre: 147388:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787848531/real 1787848531] req@00000000eec7e6c7 x1874694949079424/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787848547 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4397.122555] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4398.149817] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4401.278819] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 4401.285228] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4401.593651] Lustre: Failing over lustre-MDT0000 [ 4401.869727] Lustre: server umount lustre-MDT0000 complete [ 4402.658141] Lustre: 147390:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787848536/real 1787848536] req@00000000b75fc9a4 x1874694949080576/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787848552 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4402.675264] Lustre: 147390:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4403.176181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4403.182766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4406.754976] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4406.779602] Lustre: Skipped 1 previous similar message [ 4407.361453] LustreError: 151425:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.376225] LustreError: 151425:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 6 previous similar messages [ 4407.775676] Lustre: 147390:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787848541/real 1787848541] req@000000008a360d43 x1874694949081728/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787848557 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4411.873672] LustreError: 150307:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4411.902719] LustreError: 150307:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 4416.996036] LustreError: 150048:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4417.025503] LustreError: 150048:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 4420.310273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4420.539236] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4420.802604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4420.849893] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4423.752744] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4425.040267] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4426.234954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4426.244221] Lustre: Skipped 1 previous similar message [ 4426.260851] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4426.297856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4436.449960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4436.459152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4436.479526] Lustre: Skipped 1 previous similar message [ 4439.511530] Lustre: server umount lustre-OST0000 complete [ 4446.692369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.702495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4446.717621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4446.732862] Lustre: Skipped 2 previous similar messages [ 4449.685693] Lustre: server umount lustre-MDT0000 complete [ 4451.811628] LustreError: 150048:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.830034] LustreError: 150048:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 4453.427869] LustreError: 149154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848604 with bad export cookie 13686229514923788552 [ 4453.433789] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4453.444470] LustreError: 149154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4453.897289] Lustre: server umount lustre-MDT0001 complete [ 4465.249552] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4467.842970] Key type lgssc unregistered [ 4468.078335] LNet: 154790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4468.088729] LNetError: 154790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4468.122087] LNet: Removed LNI 192.168.203.106@tcp [ 4469.103823] Key type .llcrypt unregistered [ 4469.105851] Key type ._llcrypt unregistered [ 4507.790480] Key type ._llcrypt registered [ 4507.792535] Key type .llcrypt registered [ 4507.917689] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4508.648360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4508.695896] alg: No test for adler32 (adler32-zlib) [ 4509.750371] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 4510.004970] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 4511.689180] Key type lgssc registered [ 4512.821618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4519.443143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4519.461397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4520.875426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4525.340989] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4531.170315] LustreError: 156311:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4531.196332] LustreError: 156311:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 4534.444326] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4536.288291] LustreError: 156312:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4538.110318] LustreError: 156340:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4538.117746] LustreError: 156340:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4538.122855] LustreError: 156340:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4538.356866] Lustre: server umount lustre-MDT0000 complete [ 4553.139577] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 4564.109884] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4598.397855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4598.672728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4598.693234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4598.743364] Lustre: lustre-MDT0000: new disk, initializing [ 4598.797179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4598.808774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4602.937157] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4616.081419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4616.281166] Lustre: 159644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4616.364334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4616.368531] Lustre: Skipped 1 previous similar message [ 4616.457824] Lustre: lustre-MDT0001: new disk, initializing [ 4616.575771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4616.659075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4616.675130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4621.876775] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4632.338050] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4638.230957] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4644.985527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4645.266724] Lustre: lustre-OST0000: new disk, initializing [ 4645.275756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4645.282457] Lustre: 161613:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4645.430572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4651.622699] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4654.146916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4654.164473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4654.280429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4661.597081] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4665.526202] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4667.351908] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 12:40:17 (1787848817) [ 4941.281283] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4941.284225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4941.296609] Lustre: Skipped 2 previous similar messages [ 4941.316798] Lustre: Skipped 2 previous similar messages [ 4946.100781] Lustre: server umount lustre-MDT0000 complete [ 4946.405983] LustreError: 159651:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4946.426635] LustreError: 159651:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 4949.791674] LustreError: 160439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849100 with bad export cookie 6172589944475315747 [ 4949.793578] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4949.800708] LustreError: 160439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4951.519692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4951.524693] LustreError: 159649:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4951.545051] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4954.533555] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4959.713172] LustreError: 159649:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4959.714045] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4959.725113] LustreError: 159649:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 4964.319399] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4964.682433] Lustre: server umount lustre-MDT0001 complete [ 4974.446540] Lustre: server umount lustre-OST0000 complete [ 4984.350708] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 4995.135422] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5028.008064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5028.227162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5028.248276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5028.329314] Lustre: lustre-MDT0000: new disk, initializing [ 5028.413180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5028.433198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5034.085117] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5046.387543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.484780] Lustre: 166279:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5046.532209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5046.537589] Lustre: Skipped 1 previous similar message [ 5046.631993] Lustre: lustre-MDT0001: new disk, initializing [ 5046.763271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5046.846867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5046.860182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5051.248143] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5061.521904] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5067.749498] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5074.276274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5074.574970] Lustre: lustre-OST0000: new disk, initializing [ 5074.579531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5074.587511] Lustre: 168248:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5074.681502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5081.020745] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5083.739659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5083.753404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5083.913500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5091.197609] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5095.895975] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5096.176476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5101.192472] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5101.415478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5104.608330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5104.620783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5104.634267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5104.664067] Lustre: Skipped 1 previous similar message [ 5104.670912] Lustre: Skipped 1 previous similar message [ 5109.731403] LustreError: 168231:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5109.745494] LustreError: 168231:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 5109.827265] Lustre: server umount lustre-OST0000 complete [ 5114.848773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.856446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5114.868021] Lustre: Skipped 1 previous similar message [ 5114.888717] Lustre: Skipped 2 previous similar messages [ 5119.885172] Lustre: server umount lustre-MDT0000 complete [ 5123.093542] LustreError: 167084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849274 with bad export cookie 6172589944475474577 [ 5123.097152] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5123.110108] LustreError: 167084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5123.393799] Lustre: server umount lustre-MDT0001 complete [ 5131.366688] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:48:01 (1787849281) [ 5149.750788] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 5159.123387] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5194.487161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5194.926798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5195.000538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5195.140806] Lustre: lustre-MDT0000: new disk, initializing [ 5195.275779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5195.294586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5200.083798] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5210.161834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5210.264996] Lustre: 173002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5210.356354] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5210.511191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5214.825397] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5224.052349] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5230.177207] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5237.303097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5237.595485] Lustre: lustre-OST0000: new disk, initializing [ 5237.599303] Lustre: Skipped 1 previous similar message [ 5237.607248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5237.612909] Lustre: Skipped 2 previous similar messages [ 5237.619795] Lustre: 174979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5243.991954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5244.002484] Lustre: Skipped 1 previous similar message [ 5244.009284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5244.167354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5244.395925] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5254.609088] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5264.871258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5264.880205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5264.884402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5264.888988] LustreError: Skipped 1 previous similar message [ 5264.929133] Lustre: Skipped 1 previous similar message [ 5269.091760] Lustre: server umount lustre-OST0000 complete [ 5275.105366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.125658] Lustre: Skipped 1 previous similar message [ 5287.391584] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5287.739558] Lustre: server umount lustre-MDT0000 complete [ 5290.468792] LustreError: 173014:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.486630] LustreError: 173014:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 5291.493216] LustreError: 173802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849442 with bad export cookie 6172589944475475459 [ 5291.504684] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5291.515186] LustreError: 173802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5292.167475] Lustre: server umount lustre-MDT0001 complete [ 5300.982878] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:50:51 (1787849451) [ 5316.880550] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 5326.305722] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5364.422354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5364.840874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5364.864770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5364.931228] Lustre: lustre-MDT0000: new disk, initializing [ 5365.004632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5365.009128] Lustre: Skipped 2 previous similar messages [ 5365.028283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5371.727212] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5385.640799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.703427] Lustre: 179310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5385.928460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5391.010310] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5403.965770] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5411.642957] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5419.585320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5419.868323] Lustre: 181288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5425.508451] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5430.353791] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5430.473543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5434.580224] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5446.118371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5446.128872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5446.133435] LustreError: Skipped 2 previous similar messages [ 5446.134843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5446.157108] Lustre: Skipped 1 previous similar message [ 5446.183798] Lustre: Skipped 8 previous similar messages [ 5447.857855] Lustre: server umount lustre-OST0000 complete [ 5459.347449] Lustre: server umount lustre-MDT0000 complete [ 5461.480964] LustreError: 180105:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5461.510565] LustreError: 180105:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 5463.058454] LustreError: 179301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849614 with bad export cookie 6172589944475476544 [ 5463.059474] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5463.074683] LustreError: 179301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5463.300742] Lustre: server umount lustre-MDT0001 complete [ 5472.777414] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5475.323451] Key type lgssc unregistered [ 5475.571659] LNet: 183146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5475.586424] LNetError: 183146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5475.607152] LNet: Removed LNI 192.168.203.106@tcp [ 5476.393245] Key type .llcrypt unregistered [ 5476.395305] Key type ._llcrypt unregistered [ 5498.123675] Key type ._llcrypt registered [ 5498.126335] Key type .llcrypt registered [ 5498.253860] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5499.473927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5499.520782] alg: No test for adler32 (adler32-zlib) [ 5500.537833] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 5500.778301] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5502.415240] Key type lgssc registered [ 5503.466580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5510.685486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5510.698505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5512.105378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5516.457388] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5522.409558] LustreError: 184562:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5522.424484] LustreError: 184562:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 5525.196352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.622388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5530.758907] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5539.122884] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5544.595656] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5551.087637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5551.578786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5557.069599] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5557.743750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5564.051022] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5588.452423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5588.459107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5588.476174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5590.496202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5590.499911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5590.507068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5593.731734] Lustre: server umount lustre-OST0000 complete [ 5597.153282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5597.168508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5597.185506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5597.191419] Lustre: Skipped 1 previous similar message [ 5600.739450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5600.750150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5600.781210] Lustre: Skipped 1 previous similar message [ 5603.328545] Lustre: server umount lustre-MDT0000 complete [ 5605.859083] LustreError: 184557:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5605.877314] LustreError: 184557:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 5606.816627] LustreError: 184543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849758 with bad export cookie 2590976112445448444 [ 5606.823624] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5606.834974] LustreError: 184543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5607.203496] Lustre: server umount lustre-MDT0001 complete [ 5617.474654] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5619.888536] Key type lgssc unregistered [ 5620.121294] LNet: 188118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5620.134661] LNetError: 188118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5620.150907] LNet: Removed LNI 192.168.203.106@tcp [ 5620.906488] Key type .llcrypt unregistered [ 5620.908490] Key type ._llcrypt unregistered [ 5640.322243] Key type ._llcrypt registered [ 5640.326025] Key type .llcrypt registered [ 5640.428782] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5641.120392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5641.154628] alg: No test for adler32 (adler32-zlib) [ 5642.224195] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 5642.479989] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5644.135247] Key type lgssc registered [ 5645.046509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5651.828125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5651.860981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5653.185789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5657.237183] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5663.714637] LustreError: 189516:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5663.728516] LustreError: 189516:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 5664.677227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.008590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5668.488658] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5675.314677] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5680.849485] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5686.912300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5687.191815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5692.951677] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5698.487623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5700.141702] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5705.696539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5705.711890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.736643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5706.863967] Lustre: lustre-OST0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 5708.775624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5708.777964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5708.789156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.667380] Lustre: server umount lustre-OST0000 complete [ 5718.558690] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff890985130800.ost_server_uuid 50 [ 5720.770478] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5735.650114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5735.661312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5735.671322] Lustre: Skipped 2 previous similar messages [ 5739.683494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5739.696871] Lustre: Skipped 1 previous similar message [ 5742.084282] Lustre: server umount lustre-MDT0000 complete [ 5745.120392] LustreError: 190940:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.141332] LustreError: 190940:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 5746.192137] LustreError: 189496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849897 with bad export cookie 14407180284649460224 [ 5746.217428] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5746.656697] Lustre: server umount lustre-MDT0001 complete [ 5756.239108] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:58:26 (1787849906) [ 5763.138297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5763.695568] LustreError: 193430:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5763.799536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5768.741739] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5769.199374] LustreError: 193431:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5773.284514] LustreError: 193430:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.897990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.462748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5783.586120] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5791.976782] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5797.972660] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5804.513498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5804.984352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5808.048801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5811.321789] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5819.804183] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5827.332621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5827.507482] Lustre: lustre-OST0001: new disk, initializing [ 5827.510954] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5827.516160] Lustre: 196271:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5827.610521] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5833.430544] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5836.447728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5836.457731] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5836.533462] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5843.321689] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5847.026092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.038711] Lustre: Skipped 2 previous similar messages [ 5847.041869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5847.054322] Lustre: Skipped 1 previous similar message [ 5861.344074] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5861.490149] Lustre: server umount lustre-OST0000 complete [ 5862.375221] LustreError: 195253:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5862.412709] LustreError: 195253:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 5868.168814] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5868.398829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5873.776695] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5874.002640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5879.885431] LustreError: 195254:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5879.909363] LustreError: 195254:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 9 previous similar messages [ 5893.098748] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5893.106687] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5893.116285] Lustre: Skipped 1 previous similar message [ 5893.126125] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5893.135274] Lustre: Skipped 8 previous similar messages [ 5895.896357] Lustre: server umount lustre-OST0001 complete [ 5901.282881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5901.290708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5906.487921] Lustre: server umount lustre-MDT0000 complete [ 5910.425747] LustreError: 193412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850061 with bad export cookie 14407180284649461295 [ 5910.429845] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5910.448216] LustreError: 193412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5910.947781] Lustre: server umount lustre-MDT0001 complete [ 5937.341674] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 13:01:27 (1787850087) [ 5944.461831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.926986] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5944.992788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5945.313500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5949.541512] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5958.729317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5958.821668] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5958.860318] Lustre: 199720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5963.827182] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5971.263326] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5977.076288] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5983.906116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5984.076282] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5992.541439] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5992.739899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6002.202668] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6009.115961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6009.200343] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6009.555917] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6009.562232] Lustre: Skipped 2 previous similar messages [ 6015.859995] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6023.963852] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6028.769776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6028.780944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.792976] Lustre: Skipped 1 previous similar message [ 6028.798313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6028.802611] Lustre: Skipped 4 previous similar messages [ 6033.208550] Lustre: server umount lustre-OST0000 complete [ 6034.413936] LustreError: 201573:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6034.441852] LustreError: 201573:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 10 previous similar messages [ 6049.763332] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6049.775393] LustreError: Skipped 1 previous similar message [ 6055.960687] Lustre: server umount lustre-OST0001 complete [ 6074.335121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6074.546262] Lustre: server umount lustre-MDT0000 complete [ 6078.840205] LustreError: 201604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850230 with bad export cookie 14407180284649462702 [ 6078.842868] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6078.848583] LustreError: 201604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6079.220341] Lustre: server umount lustre-MDT0001 complete [ 6102.786793] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 13:04:13 (1787850253) [ 6120.734986] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 6133.249430] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6175.807487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6176.124917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6176.194649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6176.323287] Lustre: lustre-MDT0000: new disk, initializing [ 6176.484174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6176.509475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6182.700480] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6194.901646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6194.986687] Lustre: 207819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6195.029526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6195.033202] Lustre: Skipped 1 previous similar message [ 6195.088640] Lustre: lustre-MDT0001: new disk, initializing [ 6195.198831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6195.207415] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6199.455623] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6208.456636] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6214.302115] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6220.577685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6220.841354] Lustre: lustre-OST0000: new disk, initializing [ 6220.844744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6220.849922] Lustre: 209789:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6227.111280] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6229.073316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6229.081676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6229.178637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6235.992582] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6240.810292] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6240.991919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6245.116700] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6245.345422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6249.961948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6249.963375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6249.966749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6249.978080] Lustre: Skipped 6 previous similar messages [ 6250.004552] Lustre: Skipped 14 previous similar messages [ 6253.411778] Lustre: server umount lustre-OST0000 complete [ 6262.822371] Lustre: server umount lustre-MDT0000 complete [ 6265.312658] LustreError: 207828:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6265.333688] LustreError: 207828:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 14 previous similar messages [ 6267.243666] LustreError: 207813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850418 with bad export cookie 14407180284649464263 [ 6267.247710] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6267.254645] LustreError: 207813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6267.604987] Lustre: server umount lustre-MDT0001 complete [ 6277.161731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6283.266793] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6292.236922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6297.686598] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6305.665923] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6312.070955] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6319.089405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6325.293848] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6333.273456] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6338.177784] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6338.385545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6343.426578] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6343.714662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6348.258253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6348.265872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6348.276172] Lustre: Skipped 2 previous similar messages [ 6348.302222] LustreError: Skipped 2 previous similar messages [ 6351.964609] Lustre: server umount lustre-OST0000 complete [ 6358.468318] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6362.517545] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6392.087312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6400.261986] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6408.320799] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6410.112323] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6413.053801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.091062] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.095138] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6418.420388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6418.425039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6418.457989] Lustre: Skipped 6 previous similar messages [ 6422.599875] Lustre: server umount lustre-OST0000 complete [ 6435.654526] Lustre: server umount lustre-MDT0000 complete [ 6437.857345] LustreError: 213847:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6437.888377] LustreError: 213847:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 4 previous similar messages [ 6440.486082] LustreError: 211987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850591 with bad export cookie 14407180284649465145 [ 6440.494642] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6440.504948] LustreError: 211987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6440.733250] Lustre: server umount lustre-MDT0001 complete [ 6450.087497] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 13:10:00 (1787850600) [ 6456.281440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6456.913420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6456.920072] Lustre: Skipped 6 previous similar messages [ 6461.399430] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6469.276801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6473.878573] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6481.581290] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6486.840786] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6493.304071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6499.244541] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6506.862468] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6512.531558] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6512.935546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6519.848974] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6520.061659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6528.034114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6528.304158] Lustre: lustre-OST0001: new disk, initializing [ 6528.321662] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6528.331888] Lustre: 221344:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6534.550679] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6535.250869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6535.258485] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6535.331890] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6546.598240] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6553.408533] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6553.776175] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6561.611933] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6561.975386] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6566.376819] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6566.408763] LustreError: Skipped 2 previous similar messages [ 6566.421315] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6566.447538] Lustre: Skipped 5 previous similar messages [ 6570.840507] Lustre: server umount lustre-OST0001 complete [ 6577.030650] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6577.146396] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6581.929379] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6582.156034] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6611.292531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6617.079171] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6617.088365] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6617.099941] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6617.105687] Lustre: Skipped 1 previous similar message [ 6618.059434] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6625.901743] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6627.748209] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6636.333177] Lustre: server umount lustre-OST0001 complete [ 6660.010423] Lustre: server umount lustre-OST0000 complete [ 6670.851394] Lustre: server umount lustre-MDT0000 complete [ 6674.437264] LustreError: 217955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850825 with bad export cookie 14407180284649466111 [ 6674.448937] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6681.039733] Lustre: server umount lustre-MDT0001 complete [ 6714.409460] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 13:14:24 (1787850864) [ 6722.283104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6722.756904] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6722.794293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6728.725982] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6738.647606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6738.768109] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6738.813400] Lustre: 226594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6743.649270] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6752.039737] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6757.963799] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6765.003065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6765.325425] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6771.814544] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6779.663917] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6789.989689] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6790.053727] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6796.877436] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6805.495823] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6807.084209] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6812.582557] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6812.798195] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6817.722902] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6817.991266] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6824.345105] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8909864d6000.ost_server_uuid 50 [ 6826.212485] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8909864d6000.ost_server_uuid in IDLE state after 0 sec [ 6827.424537] Lustre: Permanently deactivating lustre-OST0001 [ 6827.440613] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6831.597195] Lustre: Permanently reactivating lustre-OST0001 [ 6831.624388] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6831.634311] Lustre: Skipped 2 previous similar messages [ 6836.712960] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6836.727281] LustreError: Skipped 4 previous similar messages [ 6836.734242] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6836.746450] Lustre: Skipped 7 previous similar messages [ 6836.752022] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6836.755864] Lustre: Skipped 12 previous similar messages [ 6842.216658] Lustre: server umount lustre-OST0001 complete [ 6845.935587] LustreError: 228443:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6845.949629] LustreError: 228443:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 38 previous similar messages [ 6853.045471] Lustre: server umount lustre-OST0000 complete [ 6863.787896] Lustre: server umount lustre-MDT0000 complete [ 6867.710916] LustreError: 226587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851018 with bad export cookie 14407180284649467399 [ 6867.727505] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6867.731947] LustreError: 226587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6868.077058] Lustre: server umount lustre-MDT0001 complete [ 6892.436144] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 13:17:22 (1787851042) [ 6900.039701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.359064] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6900.425783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6900.437607] Lustre: Skipped 2 previous similar messages [ 6904.934389] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6913.525316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6913.605026] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6913.646366] Lustre: 233289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6918.593468] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6927.408536] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6933.878703] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6940.774762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6941.084413] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6941.147936] Lustre: Permanently deactivating lustre-OST0000 [ 6941.175811] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6948.020432] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6949.401246] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6956.135874] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6963.362142] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6963.476622] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6970.275716] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6970.588430] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6977.813373] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6979.917229] Lustre: Permanently reactivating lustre-OST0000 [ 6979.930463] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6979.935845] Lustre: Skipped 2 previous similar messages [ 6986.906561] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 38 seconds. I think it's dead, and I am evicting it. exp 0000000041bf3f23, cur 1787851138 deadline 1787851130 last 1787851100 [ 6986.906686] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6986.934359] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6986.950563] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6986.960682] Lustre: Skipped 1 previous similar message [ 6987.018474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7002.561421] Lustre: server umount lustre-OST0001 complete [ 7012.547657] Lustre: server umount lustre-OST0000 complete [ 7023.060497] Lustre: server umount lustre-MDT0000 complete [ 7027.437827] LustreError: 233283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851178 with bad export cookie 14407180284649469072 [ 7027.455848] LustreError: 233283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7027.458842] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7028.049589] Lustre: server umount lustre-MDT0001 complete [ 7038.685601] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7041.340192] Key type lgssc unregistered [ 7041.639308] LNet: 238176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7041.648856] LNetError: 238176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7041.668332] LNet: Removed LNI 192.168.203.106@tcp [ 7042.486071] Key type .llcrypt unregistered [ 7042.492168] Key type ._llcrypt unregistered [ 7065.410951] Key type ._llcrypt registered [ 7065.412707] Key type .llcrypt registered [ 7065.506991] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7066.674151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7066.701200] alg: No test for adler32 (adler32-zlib) [ 7067.710238] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 7067.960761] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7069.681292] Key type lgssc registered [ 7070.880885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7081.857510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7081.867289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7083.149725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7088.653024] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7093.730856] LustreError: 239645:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7093.742567] LustreError: 239645:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 7098.466349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7098.615563] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7098.660720] Lustre: 239633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7098.678626] Lustre: Permanently deactivating lustre-MDT0001 [ 7098.696860] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7098.848225] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7098.968312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7103.198662] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7108.627727] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7111.507141] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7121.799506] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7129.833690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7130.266089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7137.749035] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7144.622724] LustreError: 241490:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7144.655673] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7146.257837] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7150.055350] LustreError: 242106:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7153.145189] LustreError: 241490:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7153.176579] LustreError: 241490:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 7153.180355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7153.600822] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7153.816990] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7159.306660] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7159.312135] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7159.800964] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7170.563751] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7175.943558] Lustre: Permanently reactivating lustre-MDT0001 [ 7175.971666] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7175.976691] Lustre: Skipped 2 previous similar messages [ 7178.883191] Lustre: lustre-MDT0001: Client 89f2eb17-112e-4f8f-9e05-2efe5be774a5 (at 192.168.203.6@tcp) reconnecting [ 7181.377676] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7181.389447] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7181.396040] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7181.417704] LustreError: 243230:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7181.418397] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7181.432759] LustreError: 243230:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7181.433909] LustreError: 243230:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.106@tcp: cfg command failed: rc = -108 [ 7181.488604] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7181.488604] [ 7181.525674] LustreError: 239636:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7186.467291] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7188.468656] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7194.541625] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7194.752599] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7199.327490] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7199.598529] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7202.542372] Lustre: Permanently deactivating lustre-MDT0001 [ 7202.563441] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7202.568432] Lustre: Skipped 2 previous similar messages [ 7208.026871] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7217.635455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7217.647094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7217.655377] Lustre: Skipped 2 previous similar messages [ 7219.379966] Lustre: server umount lustre-MDT0000 complete [ 7222.761378] LustreError: 239639:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7222.790571] LustreError: 239639:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 7223.470943] LustreError: 239624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851374 with bad export cookie 4277630792555693524 [ 7223.490249] LustreError: 239624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7223.492964] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7223.791353] Lustre: server umount lustre-MDT0001 complete [ 7229.817871] Lustre: server umount lustre-OST0000 complete [ 7241.771472] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 13:23:11 (1787851391) [ 7253.697137] Lustre: server umount lustre-OST0001 complete [ 7261.177757] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 7272.257919] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7311.978040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.173226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7312.179707] Lustre: Skipped 2 previous similar messages [ 7312.200554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7312.258981] Lustre: lustre-MDT0000: new disk, initializing [ 7312.345027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7312.374344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7317.357692] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7329.233898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7329.341896] Lustre: 247906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7329.397979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7329.402500] Lustre: Skipped 1 previous similar message [ 7329.498561] Lustre: lustre-MDT0001: new disk, initializing [ 7329.595750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7329.644341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7329.659080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7334.685390] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7343.870215] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7349.995281] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7357.248310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7357.489790] Lustre: lustre-OST0000: new disk, initializing [ 7357.493914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7357.500255] Lustre: 249872:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7357.586981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7364.069553] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7367.245158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7367.273464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7367.436177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7374.341536] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7376.923152] LustreError: 250350:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7385.847804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7386.019724] Lustre: lustre-OST0001: new disk, initializing [ 7386.025640] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7386.035079] Lustre: 250988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7387.023494] LustreError: 250350:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7387.059575] LustreError: 247912:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7392.875738] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7395.971469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7395.976891] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7396.050333] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7397.159125] LustreError: 247912:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7403.114909] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7405.098639] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7408.739743] Lustre: lustre-OST0001: Not available for connect from 192.168.203.6@tcp (stopping) [ 7411.683189] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7411.687190] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7411.702834] LustreError: Skipped 1 previous similar message [ 7411.706061] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7411.738627] Lustre: Skipped 4 previous similar messages [ 7413.782525] Lustre: server umount lustre-OST0001 complete [ 7413.827671] LustreError: 249858:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7432.160592] LustreError: 249857:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7432.165356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.168041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7432.168048] Lustre: Skipped 1 previous similar message [ 7432.184351] LustreError: 249857:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 7 previous similar messages [ 7432.222449] Lustre: Skipped 1 previous similar message [ 7437.284093] Lustre: server umount lustre-OST0000 complete [ 7442.401774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7442.401927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7442.414277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7442.432431] Lustre: Skipped 2 previous similar messages [ 7447.448492] Lustre: server umount lustre-MDT0000 complete [ 7450.973241] LustreError: 247896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851602 with bad export cookie 4277630792555696338 [ 7450.976201] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7450.984497] LustreError: 247896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7457.381503] Lustre: server umount lustre-MDT0001 complete [ 7475.328897] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7479.605417] Key type lgssc unregistered [ 7480.066516] LNet: 253038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7480.081988] LNetError: 253038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7480.101709] LNet: Removed LNI 192.168.203.106@tcp [ 7482.196788] Key type .llcrypt unregistered [ 7482.201565] Key type ._llcrypt unregistered [ 7527.047670] Key type ._llcrypt registered [ 7527.050453] Key type .llcrypt registered [ 7527.148388] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7528.715328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7528.788701] alg: No test for adler32 (adler32-zlib) [ 7529.822274] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 7530.010529] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7531.703472] Key type lgssc registered [ 7533.528741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7541.849097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7541.877333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7543.034347] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7543.057381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7543.170834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7547.295402] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7557.278908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7557.448905] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7557.514829] Lustre: 255247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7557.923574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7564.445365] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7572.859951] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7579.477359] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7586.528833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7586.794707] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7586.986960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7595.598423] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7595.893144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7606.539414] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7662.561355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7662.567547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7662.571267] LustreError: Skipped 1 previous similar message [ 7662.597231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7662.605560] Lustre: Skipped 1 previous similar message [ 7665.681157] Lustre: server umount lustre-OST0000 complete [ 7670.249746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7681.669639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7682.000443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7683.065959] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7683.087609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7683.091180] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7688.557709] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7696.058180] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7739.361635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7739.362651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7739.372929] LustreError: Skipped 1 previous similar message [ 7739.376047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7739.446422] Lustre: Skipped 2 previous similar messages [ 7741.642890] Lustre: server umount lustre-OST0000 complete [ 7747.552674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7747.564657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7747.569895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7747.581656] Lustre: Skipped 1 previous similar message [ 7749.601211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7749.601478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7749.662363] Lustre: Skipped 1 previous similar message [ 7754.721241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7754.726515] Lustre: Skipped 1 previous similar message [ 7760.351880] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7760.683678] Lustre: server umount lustre-MDT0000 complete [ 7765.537817] LustreError: 255238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851916 with bad export cookie 6567078638076084015 [ 7765.550308] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7765.550719] LustreError: 255238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7765.944265] Lustre: server umount lustre-MDT0001 complete [ 7779.498930] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7783.607398] Key type lgssc unregistered [ 7783.991929] LNet: 260674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7783.996430] LNetError: 260674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7784.010697] LNet: Removed LNI 192.168.203.106@tcp [ 7785.198247] Key type .llcrypt unregistered [ 7785.203070] Key type ._llcrypt unregistered [ 7813.149895] Key type ._llcrypt registered [ 7813.153152] Key type .llcrypt registered [ 7813.248992] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7814.486373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7814.603971] alg: No test for adler32 (adler32-zlib) [ 7815.833961] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 7816.149983] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7817.887171] Key type lgssc registered [ 7819.148397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7827.010479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7827.037121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.463549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7833.075331] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7838.696074] LustreError: 262096:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7838.724715] LustreError: 262096:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 7842.602768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7843.223699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7848.269795] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7855.922225] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7861.862902] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7869.841344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7870.405635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7873.478385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7877.695655] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7886.694354] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7914.468332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7914.470236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7914.477412] LustreError: Skipped 1 previous similar message [ 7914.489550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7919.587656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7919.596531] Lustre: Skipped 2 previous similar messages [ 7920.137505] Lustre: server umount lustre-OST0000 complete [ 7924.707340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7924.709954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7924.715523] Lustre: Skipped 2 previous similar messages [ 7924.747706] Lustre: Skipped 1 previous similar message [ 7929.825960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7929.836073] Lustre: Skipped 1 previous similar message [ 7931.198728] Lustre: server umount lustre-MDT0000 complete [ 7934.944602] LustreError: 262092:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7934.987161] LustreError: 262092:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 7937.798100] LustreError: 262077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852088 with bad export cookie 14700674587320262887 [ 7937.820880] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7937.827714] LustreError: 262077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7938.229656] Lustre: server umount lustre-MDT0001 complete [ 7949.734217] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7952.396542] Key type lgssc unregistered [ 7952.626270] LNet: 266407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7952.640880] LNetError: 266407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7953.710441] LNet: Removed LNI 192.168.203.106@tcp [ 7954.588429] Key type .llcrypt unregistered [ 7954.591381] Key type ._llcrypt unregistered [ 7973.204589] Key type ._llcrypt registered [ 7973.211668] Key type .llcrypt registered [ 7973.349234] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7974.450230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7974.582575] alg: No test for adler32 (adler32-zlib) [ 7975.784233] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 7976.238617] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7978.129331] Key type lgssc registered [ 7980.037419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7992.201816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7992.261220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7994.300975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8002.977985] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8004.578056] LustreError: 267630:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8004.593614] LustreError: 267630:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 8005.610121] LustreError: 267629:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8010.724525] LustreError: 267630:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8014.008730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8014.485463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8019.614762] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8027.448642] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8033.473193] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8039.511605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8039.858935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8040.903635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8046.076234] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8055.007553] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8074.735765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8074.745830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8074.747683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8074.755926] Lustre: Skipped 1 previous similar message [ 8076.066862] Lustre: server umount lustre-OST0000 complete [ 8080.867243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8080.875069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8080.897392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8080.915590] Lustre: Skipped 1 previous similar message [ 8084.960440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8084.967186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8090.081334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8090.092028] Lustre: Skipped 2 previous similar messages [ 8095.199134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8095.200472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8095.223551] Lustre: Skipped 1 previous similar message [ 8095.401928] Lustre: server umount lustre-MDT0000 complete [ 8099.022750] LustreError: 269473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852250 with bad export cookie 6095844091918254485 [ 8099.026393] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8099.045621] LustreError: 269473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8099.269313] Lustre: server umount lustre-MDT0001 complete [ 8111.423813] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 8115.275735] Key type lgssc unregistered [ 8115.703835] LNet: 271311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8115.715679] LNetError: 271311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8115.735396] LNet: Removed LNI 192.168.203.106@tcp [ 8116.803333] Key type .llcrypt unregistered [ 8116.811704] Key type ._llcrypt unregistered [ 8146.816925] Key type ._llcrypt registered [ 8146.823521] Key type .llcrypt registered [ 8147.125115] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8148.876413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8148.903368] alg: No test for adler32 (adler32-zlib) [ 8150.120339] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 8150.473502] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 8152.183191] Key type lgssc registered [ 8153.562390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8163.266521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8163.277322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8164.784702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8170.059364] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8174.071307] LustreError: 272731:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8174.125648] LustreError: 272731:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 8179.173670] LustreError: 272732:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8180.211180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8180.565262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8185.216528] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8194.502227] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8202.683889] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8210.937306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8211.339875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8218.614206] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8227.070896] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8233.866536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8259.558442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8259.586161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8259.587912] Lustre: Skipped 1 previous similar message [ 8259.605945] Lustre: Skipped 1 previous similar message [ 8263.966051] Lustre: server umount lustre-OST0000 complete [ 8269.803837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8269.826815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8269.828648] Lustre: Skipped 1 previous similar message [ 8269.837876] Lustre: Skipped 1 previous similar message [ 8274.915313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8274.918942] Lustre: Skipped 1 previous similar message [ 8279.008971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8279.012578] Lustre: Skipped 1 previous similar message [ 8283.103425] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8283.548191] Lustre: server umount lustre-MDT0000 complete [ 8284.132141] LustreError: 272732:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8284.148886] LustreError: 272732:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 8289.250561] LustreError: 272727:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8289.304478] LustreError: 272727:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 8289.608088] LustreError: 272711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852440 with bad export cookie 14153738798816059327 [ 8289.613918] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8289.635582] LustreError: 272711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8290.743875] Lustre: server umount lustre-MDT0001 complete [ 8307.643838] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 8310.747322] Key type lgssc unregistered [ 8311.098631] LNet: 277046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8311.103144] LNetError: 277046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8311.124499] LNet: Removed LNI 192.168.203.106@tcp [ 8311.903247] Key type .llcrypt unregistered [ 8311.905622] Key type ._llcrypt unregistered [ 8333.974896] Key type ._llcrypt registered [ 8333.976744] Key type .llcrypt registered [ 8334.091249] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8335.254772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8335.297992] alg: No test for adler32 (adler32-zlib) [ 8336.523865] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 8337.033620] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 8338.759273] Key type lgssc registered [ 8339.987686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8350.086592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8350.131504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8352.131020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8359.426995] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8362.471419] LustreError: 278284:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8362.508028] LustreError: 278284:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 8367.592086] LustreError: 278286:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8371.530073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8372.681840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8379.765849] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8387.811911] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8394.903064] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8402.039140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8402.416632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8408.841591] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8416.716646] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8421.877953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8433.121518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8433.125761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8433.135082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8437.218068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8437.218773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.221165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8438.527421] Lustre: server umount lustre-OST0000 complete [ 8444.387811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8444.413593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8444.460399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8444.467539] Lustre: Skipped 1 previous similar message [ 8447.457378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8447.459944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8449.239290] Lustre: server umount lustre-MDT0000 complete [ 8452.577975] LustreError: 278291:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8452.624603] LustreError: 278291:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 8454.468918] LustreError: 278251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852605 with bad export cookie 8183325659477732259 [ 8454.484737] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8454.490400] LustreError: 278251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8454.839942] Lustre: server umount lustre-MDT0001 complete [ 8466.736209] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 8469.306679] Key type lgssc unregistered [ 8469.609964] LNet: 281990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8469.620240] LNetError: 281990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8469.644891] LNet: Removed LNI 192.168.203.106@tcp [ 8470.518836] Key type .llcrypt unregistered [ 8470.520474] Key type ._llcrypt unregistered [ 8484.281181] Key type ._llcrypt registered [ 8484.283493] Key type .llcrypt registered [ 8484.374723] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8505.973272] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 8523.163459] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8525.172919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8525.217090] alg: No test for adler32 (adler32-zlib) [ 8526.415074] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 8526.763170] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 8528.375153] Key type lgssc registered [ 8529.931950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8572.724856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8572.753682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8573.990752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8574.037931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8574.107341] Lustre: lustre-MDT0000: new disk, initializing [ 8574.147428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8574.156305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8579.322931] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8592.539951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8592.786128] Lustre: 285545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8592.837488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8592.840962] Lustre: Skipped 1 previous similar message [ 8592.893429] Lustre: lustre-MDT0001: new disk, initializing [ 8592.977896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8593.005072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8593.015976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8599.533914] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8611.497303] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8618.346929] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8626.537097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8626.834876] Lustre: lustre-OST0000: new disk, initializing [ 8626.838111] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8626.846985] Lustre: 287518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8626.933397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8633.478524] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8635.492923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8635.502256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8635.597497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8644.045366] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8649.718930] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8650.065546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8654.726956] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8654.916661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8661.474676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8661.484165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8661.498099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8661.501065] Lustre: Skipped 1 previous similar message [ 8666.597347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8666.614221] Lustre: Skipped 1 previous similar message [ 8671.199619] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8671.568706] Lustre: server umount lustre-OST0000 complete [ 8676.505968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.534557] Lustre: Skipped 1 previous similar message [ 8676.560170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8676.576753] Lustre: Skipped 1 previous similar message [ 8681.954417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8681.968346] Lustre: Skipped 1 previous similar message [ 8682.904747] Lustre: server umount lustre-MDT0000 complete [ 8687.007294] LustreError: 285540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852838 with bad export cookie 3238389328326956773 [ 8687.017603] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8687.022623] LustreError: 285540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8687.415798] Lustre: server umount lustre-MDT0001 complete [ 8696.884177] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:47:27 (1787852847) [ 8703.928431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8704.630571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8708.837243] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8710.112960] LustreError: 289931:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8710.138204] LustreError: 289931:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 8714.212645] LustreError: 289932:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8717.798903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8718.211783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8723.147763] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8736.081510] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8745.808730] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8756.883219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8757.235392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8763.823472] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8772.188079] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8790.007266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8790.009375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8790.019877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8790.035297] Lustre: Skipped 2 previous similar messages [ 8790.053891] LustreError: Skipped 1 previous similar message [ 8791.460523] Lustre: server umount lustre-OST0000 complete [ 8800.229313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8800.233914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8800.255254] Lustre: Skipped 1 previous similar message [ 8800.271356] Lustre: Skipped 2 previous similar messages [ 8802.020344] Lustre: server umount lustre-MDT0000 complete [ 8805.347434] LustreError: 289932:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8805.365416] LustreError: 289932:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 8806.209276] LustreError: 289914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852957 with bad export cookie 3238389328326957655 [ 8806.210887] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8806.217829] LustreError: 289914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8806.744266] Lustre: server umount lustre-MDT0001 complete [ 8818.734884] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 8821.483325] Key type lgssc unregistered [ 8821.766929] LNet: 293491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8821.781649] LNetError: 293491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8821.815378] LNet: Removed LNI 192.168.203.106@tcp [ 8822.720156] Key type .llcrypt unregistered [ 8822.723044] Key type ._llcrypt unregistered [ 8850.387794] Key type ._llcrypt registered [ 8850.391765] Key type .llcrypt registered [ 8850.537710] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 8865.111823] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8866.421473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8866.601530] alg: No test for adler32 (adler32-zlib) [ 8867.793645] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 8868.027337] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 8869.807180] Key type lgssc registered [ 8870.921812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8915.658941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8915.703250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8917.036814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8917.078824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8917.199988] Lustre: lustre-MDT0000: new disk, initializing [ 8917.348618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8917.377461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8922.884979] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8933.502526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8933.669263] Lustre: 297315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8933.741576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8933.751037] Lustre: Skipped 1 previous similar message [ 8933.834293] Lustre: lustre-MDT0001: new disk, initializing [ 8933.976268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8934.056591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8934.067895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8938.971486] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8948.995924] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8954.594694] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8961.567807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8961.794896] Lustre: lustre-OST2710: new disk, initializing [ 8961.799077] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8961.805982] Lustre: 299286:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8961.898839] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8968.668532] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8970.298836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8970.315586] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8970.487747] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8977.338034] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8983.617797] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8983.787730] Lustre: lustre-OST03e8: new disk, initializing [ 8983.792676] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8983.808345] Lustre: 300351:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8983.959733] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8990.220040] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8993.906738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8993.924054] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8994.055308] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8997.333782] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9003.801993] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9004.051833] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9008.567657] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9008.763779] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9014.859698] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9015.274484] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9021.567489] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9021.853775] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9029.008285] Lustre: server umount lustre-MDT0000 complete [ 9030.115580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9030.144957] Lustre: Skipped 3 previous similar messages [ 9032.238642] LustreError: 300349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853183 with bad export cookie 1716616613995354087 [ 9032.243252] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9032.254559] LustreError: 300349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9032.900137] Lustre: server umount lustre-MDT0001 complete [ 9047.733328] Lustre: server umount lustre-OST2710 complete [ 9051.679187] Lustre: 295113:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853186/real 1787853186] req@0000000058c565a8 x1874699767317376/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787853202 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9051.715729] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9052.020903] Lustre: server umount lustre-OST03e8 complete [ 9072.900878] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 9086.323867] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 9124.259339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9124.488756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9124.513424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9124.599833] Lustre: lustre-MDT0000: new disk, initializing [ 9124.783806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9124.806261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9129.466316] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9140.644829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9140.808838] Lustre: 305598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9140.899352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9140.906610] Lustre: Skipped 1 previous similar message [ 9140.995866] Lustre: lustre-MDT0001: new disk, initializing [ 9141.121625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9141.177121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9141.187223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9146.060370] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9155.345824] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9161.660659] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9168.247347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9168.496577] Lustre: lustre-OST0000: new disk, initializing [ 9168.501703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9168.509774] Lustre: 307564:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9168.639046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9175.043506] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9178.727665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9178.740315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9178.845080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9185.108982] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9189.841869] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9190.149797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9194.771759] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9194.934145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9199.590041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9199.590297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9199.597870] LustreError: Skipped 1 previous similar message [ 9199.618490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9199.628660] Lustre: Skipped 1 previous similar message [ 9203.037948] Lustre: server umount lustre-OST0000 complete [ 9207.776664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9207.791425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9207.802593] Lustre: Skipped 1 previous similar message [ 9207.808036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9209.825605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9209.832577] Lustre: Skipped 1 previous similar message [ 9212.839985] Lustre: server umount lustre-MDT0000 complete [ 9214.944895] LustreError: 306784:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9214.980923] LustreError: 306784:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 6 previous similar messages [ 9216.515504] LustreError: 305589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853367 with bad export cookie 1716616613995355774 [ 9216.522850] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9216.530021] LustreError: 305589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9216.914853] Lustre: server umount lustre-MDT0001 complete [ 9225.810405] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:56:16 (1787853376) [ 9227.499257] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9229.170377] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:56:19 (1787853379) [ 9232.663435] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 9259.151679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9259.387341] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9259.403915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9259.552389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9264.174543] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9272.728314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9272.824393] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9272.857992] Lustre: 311426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9277.582614] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9299.112288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9299.350509] Lustre: Denying initial registration attempt from nid 192.168.203.106@tcp, specified as failover [ 9299.361445] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9299.366597] LustreError: 312880:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9299.371651] LustreError: 312880:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9299.375811] LustreError: 312880:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9299.473785] Lustre: server umount lustre-OST0000 complete [ 9299.480267] LustreError: 312880:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9309.153104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9309.154373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9309.157183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9309.179917] Lustre: Skipped 2 previous similar messages [ 9312.044112] Lustre: server umount lustre-MDT0000 complete [ 9314.273263] LustreError: 311432:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9314.310131] LustreError: 311432:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 9315.735636] LustreError: 311418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853466 with bad export cookie 1716616613995356656 [ 9315.751778] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9315.752717] LustreError: 311418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9316.123442] Lustre: server umount lustre-MDT0001 complete [ 9326.687463] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 9329.653820] Key type lgssc unregistered [ 9330.029727] LNet: 314003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9330.036380] LNetError: 314003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9330.054199] LNet: Removed LNI 192.168.203.106@tcp [ 9331.016766] Key type .llcrypt unregistered [ 9331.018579] Key type ._llcrypt unregistered [ 9342.674273] Key type ._llcrypt registered [ 9342.679217] Key type .llcrypt registered [ 9342.790229] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 9344.039776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9344.121569] alg: No test for adler32 (adler32-zlib) [ 9345.189780] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 9345.471381] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 9347.183651] Key type lgssc registered [ 9348.309605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9395.086977] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 9403.243551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9403.269040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9404.451822] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9404.486452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9404.644914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9409.240959] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9418.569430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9418.660195] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9418.678860] Lustre: 316769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9418.993714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9423.074080] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9431.023564] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9436.466563] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9456.688338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9456.888771] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9457.098532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9462.997904] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9470.030884] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9472.484884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9472.490562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9472.496288] Lustre: Skipped 1 previous similar message [ 9472.517722] Lustre: Skipped 1 previous similar message [ 9477.606773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9477.613299] Lustre: Skipped 1 previous similar message [ 9478.378538] Lustre: server umount lustre-OST0000 complete [ 9482.724643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9482.732026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9482.740904] Lustre: Skipped 1 previous similar message [ 9482.751838] Lustre: Skipped 1 previous similar message [ 9487.843032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9487.856556] Lustre: Skipped 1 previous similar message [ 9488.318240] Lustre: server umount lustre-MDT0000 complete [ 9492.205780] LustreError: 316762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853643 with bad export cookie 11721468801624883010 [ 9492.214993] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9492.222663] LustreError: 316762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9492.612896] Lustre: server umount lustre-MDT0001 complete [ 9502.236352] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 9504.931798] Key type lgssc unregistered [ 9505.209685] LNet: 320347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9505.225651] LNetError: 320347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9505.248228] LNet: Removed LNI 192.168.203.106@tcp [ 9506.006557] Key type .llcrypt unregistered [ 9506.008740] Key type ._llcrypt unregistered [ 9529.741646] Key type ._llcrypt registered [ 9529.746533] Key type .llcrypt registered [ 9529.881886] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 9531.044189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9531.087422] alg: No test for adler32 (adler32-zlib) [ 9532.177987] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 9532.459078] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 9534.159680] Key type lgssc registered [ 9535.355224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9543.032833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9543.055931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9544.499207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9548.722099] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9553.890783] LustreError: 321781:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9553.903536] LustreError: 321781:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 9557.640246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9558.081307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9562.531887] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9570.234801] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9576.549445] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9584.946113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9585.379560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9591.867650] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9599.576203] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9617.505795] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 9619.429489] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9619.433922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9619.435988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9619.457739] Lustre: Skipped 2 previous similar messages [ 9619.490628] Lustre: Skipped 2 previous similar messages [ 9621.036587] Lustre: server umount lustre-MDT0000 complete [ 9622.605037] LustreError: 321777:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9624.480648] LustreError: 321762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853775 with bad export cookie 15509984910645269460 [ 9624.483858] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9624.496692] LustreError: 321762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9624.543969] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9624.557469] LustreError: 323825:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9624.564559] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9627.718290] LustreError: 321775:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9627.757884] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.6@tcp (stopping) [ 9630.947703] Lustre: server umount lustre-MDT0001 complete [ 9637.258157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9645.307392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9649.120019] LustreError: 321089:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000499baa54 x1874700463546624/t0(0) o250->MGC192.168.203.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9649.395988] LustreError: 323612:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9649.432517] LustreError: 323612:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 9649.468499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9649.534199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9653.667305] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9659.749820] LustreError: 325396:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9659.761656] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9659.774980] LustreError: 325396:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 9662.835320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9663.284117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9668.144207] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9668.598039] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9676.480570] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9678.145256] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9683.702677] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9685.330069] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9690.593549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9690.600243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9690.615793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9690.619856] Lustre: Skipped 1 previous similar message [ 9694.178316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9694.189188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9696.375501] Lustre: server umount lustre-OST0000 complete [ 9704.419635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9704.422128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9704.432738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9704.443976] Lustre: Skipped 3 previous similar messages [ 9706.414322] Lustre: server umount lustre-MDT0000 complete [ 9709.538613] LustreError: 325394:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9709.559565] LustreError: 325394:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 9710.270611] LustreError: 321762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853861 with bad export cookie 15509984910645277027 [ 9710.279894] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9710.296225] LustreError: 321762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9710.667968] Lustre: server umount lustre-MDT0001 complete [ 9724.516784] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 9727.608784] Key type lgssc unregistered [ 9727.920668] LNet: 328071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9727.923786] LNetError: 328071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9727.946141] LNet: Removed LNI 192.168.203.106@tcp [ 9728.861315] Key type .llcrypt unregistered [ 9728.872103] Key type ._llcrypt unregistered [ 9753.156904] Key type ._llcrypt registered [ 9753.161660] Key type .llcrypt registered [ 9753.275583] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 9754.238558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9754.314904] alg: No test for adler32 (adler32-zlib) [ 9755.415279] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 9755.727209] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 9757.447161] Key type lgssc registered [ 9758.677194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9766.571734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9766.589579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9768.079332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9772.100659] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9778.659758] LustreError: 329489:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9778.692790] LustreError: 329489:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 9778.965820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9779.279980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9783.015469] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9789.085908] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9793.966955] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9798.629167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9798.633507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9798.641900] Lustre: Skipped 1 previous similar message [ 9803.687521] Lustre: server umount lustre-MDT0000 complete [ 9803.744397] LustreError: 329490:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9803.759420] LustreError: 329490:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 9804.770705] LustreError: 329485:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9804.790208] LustreError: 329485:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 9807.119114] LustreError: 329471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853958 with bad export cookie 17553103253593361368 [ 9807.137163] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9807.445822] Lustre: server umount lustre-MDT0001 complete [ 9821.252781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9821.604036] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9821.653386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9821.847284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9826.126330] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9833.288233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9833.365126] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9833.399707] Lustre: 332076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9833.619885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9837.577697] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9843.449244] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9848.426122] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9857.641606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9857.831288] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9857.837969] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9857.845129] LustreError: 333962:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9857.854357] LustreError: 333962:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9857.863262] LustreError: 333962:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9857.915224] Lustre: server umount lustre-OST0000 complete [ 9857.919267] LustreError: 333962:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9868.460447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9868.762283] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9869.021653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9874.557641] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9877.642320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9881.117742] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9890.318944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9890.357330] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9890.403454] Lustre: lustre-OST0001: new disk, initializing [ 9890.407727] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9890.413371] Lustre: 335505:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9890.483293] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9897.910446] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9898.726212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9898.737791] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9898.871352] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9906.127672] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9909.224913] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9909.226646] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9909.245698] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9909.249092] LustreError: Skipped 1 previous similar message [ 9909.266327] Lustre: Skipped 2 previous similar messages [ 9914.345607] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9914.354514] Lustre: Skipped 1 previous similar message [ 9919.474828] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9919.485818] Lustre: Skipped 1 previous similar message [ 9923.039736] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9923.116590] Lustre: server umount lustre-OST0001 complete [ 9924.586665] LustreError: 334427:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.602334] LustreError: 334427:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 9928.672288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9928.679818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9928.690959] Lustre: Skipped 1 previous similar message [ 9928.694472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9929.703893] LustreError: 334429:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9929.730354] LustreError: 334429:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 9932.367804] Lustre: server umount lustre-OST0000 complete [ 9936.352239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9936.361524] LustreError: Skipped 1 previous similar message [ 9936.364483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9936.376402] Lustre: Skipped 1 previous similar message [ 9939.938939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9939.952552] Lustre: Skipped 4 previous similar messages [ 9941.814939] Lustre: server umount lustre-MDT0000 complete [ 9944.856376] LustreError: 332068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854096 with bad export cookie 17553103253593361921 [ 9944.860597] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9944.868554] LustreError: 332068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9945.199359] Lustre: server umount lustre-MDT0001 complete [ 9954.840420] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 9957.594827] Key type lgssc unregistered [ 9957.923949] LNet: 337467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9957.932793] LNetError: 337467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9957.959293] LNet: Removed LNI 192.168.203.106@tcp [ 9958.852821] Key type .llcrypt unregistered [ 9958.855472] Key type ._llcrypt unregistered [10024.118312] Key type ._llcrypt registered [10024.119935] Key type .llcrypt registered [10024.231417] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [10037.892565] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [10039.101467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10039.219952] alg: No test for adler32 (adler32-zlib) [10040.247537] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [10040.522089] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [10042.167174] Key type lgssc registered [10043.172495] Lustre: Echo OBD driver; http://www.lustre.org/ [10076.128142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10076.156580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10077.437379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10077.490573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10077.649232] Lustre: lustre-MDT0000: new disk, initializing [10077.843427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10077.875479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10083.843972] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10094.791682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10094.915557] Lustre: 342392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10094.963803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10094.971108] Lustre: Skipped 1 previous similar message [10095.039280] Lustre: lustre-MDT0001: new disk, initializing [10095.202146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10095.250124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10095.263218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10099.979086] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10109.055424] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10114.561328] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10120.987641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10121.220168] Lustre: lustre-OST0000: new disk, initializing [10121.224122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10121.230234] Lustre: 344358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10121.308934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10126.568202] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10127.421332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10127.432548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10127.524749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10135.595707] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10139.523692] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10139.684923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10144.232714] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10144.475329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10148.320680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10148.322798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10148.331901] LustreError: Skipped 1 previous similar message [10148.352774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10148.366654] Lustre: Skipped 1 previous similar message [10152.574276] Lustre: server umount lustre-OST0000 complete [10156.512289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10156.518480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10156.527422] Lustre: Skipped 1 previous similar message [10156.532160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10158.560192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10158.562346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10162.701288] Lustre: server umount lustre-MDT0000 complete [10166.154528] LustreError: 342382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854317 with bad export cookie 17374483247270915076 [10166.163356] LustreError: 342382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10166.165135] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10166.439870] Lustre: server umount lustre-MDT0001 complete [10176.064977] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 14:12:06 (1787854326) [10185.178398] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 14:12:15 (1787854335) [10195.417460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10195.822799] LustreError: 347115:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10195.837807] LustreError: 347115:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [10195.886372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10200.018318] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10201.059794] LustreError: 347116:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10206.178100] LustreError: 347115:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10207.877158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10208.386357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10212.304596] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10219.091920] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10224.051695] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10230.005951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10230.313086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10235.576487] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10242.558463] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10250.336695] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10251.886388] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10253.802470] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10255.311398] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10256.929447] Lustre: Failing over lustre-MDT0000 [10257.169611] Lustre: server umount lustre-MDT0000 complete [10259.425380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10259.434995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10259.460443] LustreError: 347844:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10265.669241] LustreError: 347111:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10265.691691] LustreError: 347111:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 4 previous similar messages [10275.590129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10275.696973] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10275.929431] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (not set up) [10275.945677] Lustre: Skipped 1 previous similar message [10276.113364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10276.191848] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10281.044313] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10281.453886] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10281.467446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10281.518111] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10281.597355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10291.393957] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10293.416811] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10297.220988] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10299.255114] Lustre: Failing over lustre-MDT0000 [10299.642188] Lustre: server umount lustre-MDT0000 complete [10301.530600] LustreError: 349421:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10301.538971] LustreError: 349421:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 7 previous similar messages [10301.920588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10301.963660] Lustre: Skipped 4 previous similar messages [10317.972503] LustreError: 347110:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10318.008527] LustreError: 347110:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 15 previous similar messages [10318.114651] Lustre: 340616:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854453/real 1787854453] req@000000009b40b626 x1874700996270464/t0(0) o400->MGC192.168.203.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787854469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10318.158783] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10319.361087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10328.551377] LustreError: 340613:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f40787d5 x1874700996276736/t0(0) o250->MGC192.168.203.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10328.860307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10328.916527] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10333.259097] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10333.757925] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10334.186575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10334.193436] Lustre: Skipped 2 previous similar messages [10334.218964] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10334.266060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10343.286665] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10345.030945] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10354.669197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10354.676162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10354.676926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10354.701918] Lustre: Skipped 1 previous similar message [10354.732548] Lustre: Skipped 2 previous similar messages [10357.057564] Lustre: server umount lustre-MDT0000 complete [10359.796927] LustreError: 347115:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10359.818770] LustreError: 347115:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 13 previous similar messages [10360.468123] LustreError: 347096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854511 with bad export cookie 17374483247270917792 [10360.494725] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10360.842943] Lustre: server umount lustre-MDT0001 complete [10375.107819] Lustre: server umount lustre-OST0000 complete [10391.477117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10396.934451] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10405.686336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10406.208417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10406.222307] Lustre: Skipped 1 previous similar message [10411.770694] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10420.859585] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10426.953399] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10434.069562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10437.493478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10440.639593] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10447.947144] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10457.056351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10457.066593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10457.069614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10457.081610] Lustre: Skipped 2 previous similar messages [10457.094611] Lustre: Skipped 1 previous similar message [10471.391101] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10471.639606] Lustre: server umount lustre-OST0000 complete [10477.544377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10477.549342] Lustre: Skipped 5 previous similar messages [10490.335145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10490.612643] Lustre: server umount lustre-MDT0000 complete [10492.900302] LustreError: 356041:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10492.926527] LustreError: 356041:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 4 previous similar messages [10494.593961] LustreError: 353444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854645 with bad export cookie 17374483247270918534 [10494.605682] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10494.608783] LustreError: 353444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10494.950406] Lustre: server umount lustre-MDT0001 complete [10506.864695] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [10509.798586] Key type lgssc unregistered [10510.090820] LNet: 357002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10510.097756] LNetError: 357002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10510.109173] LNet: Removed LNI 192.168.203.106@tcp [10510.884709] Key type .llcrypt unregistered [10510.886609] Key type ._llcrypt unregistered [10543.781378] Key type ._llcrypt registered [10543.786859] Key type .llcrypt registered [10543.933493] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [10544.702415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10544.739923] alg: No test for adler32 (adler32-zlib) [10545.938662] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [10546.301439] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [10548.015260] Key type lgssc registered [10549.451191] Lustre: Echo OBD driver; http://www.lustre.org/ [10558.428988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10558.443267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10559.927905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10565.462162] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10570.222873] LustreError: 358519:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10570.247923] LustreError: 358519:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [10574.670986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10575.072235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10579.179987] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10588.125946] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10595.843850] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10603.903350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10604.180624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10605.244581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10610.298657] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10618.777634] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10631.140073] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10631.154600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10631.155860] Lustre: Skipped 2 previous similar messages [10631.163441] Lustre: Skipped 2 previous similar messages [10631.995633] Lustre: server umount lustre-MDT0000 complete [10635.464964] LustreError: 360368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854786 with bad export cookie 7599470275321005594 [10635.465739] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10635.477988] LustreError: 360368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10635.779838] Lustre: server umount lustre-MDT0001 complete [10649.866251] Lustre: server umount lustre-OST0000 complete [10663.753150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10664.353392] LustreError: 362234:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10664.490273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10669.305912] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10669.539787] LustreError: 362235:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10674.659116] LustreError: 362234:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10678.264065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10678.644309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10683.639578] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10691.437201] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10697.562420] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10705.466991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10705.846950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10708.918773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10712.633968] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10720.399269] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10733.540393] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10733.546630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10733.566821] Lustre: Skipped 2 previous similar messages [10733.586574] Lustre: Skipped 2 previous similar messages [10738.673568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10738.684510] Lustre: Skipped 2 previous similar messages [10739.856232] Lustre: server umount lustre-MDT0000 complete [10743.782305] LustreError: 363474:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10743.786744] LustreError: 362214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854894 with bad export cookie 7599470275321006714 [10743.787900] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10743.794144] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10743.816723] LustreError: 363474:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [10743.819871] LustreError: 362214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10743.824269] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10744.448763] Lustre: server umount lustre-MDT0001 complete [10751.589602] Lustre: server umount lustre-OST0000 complete [10767.886975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10768.244925] LustreError: 366144:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10768.295546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10773.347395] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10783.087439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10788.324973] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10795.946899] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10802.887988] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10810.492176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10810.999969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10811.016358] Lustre: Skipped 1 previous similar message [10813.038937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10817.714056] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10826.134218] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10833.889207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10833.891026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10833.896129] LustreError: Skipped 1 previous similar message [10833.898177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10833.926946] Lustre: Skipped 1 previous similar message [10837.454930] Lustre: server umount lustre-OST0000 complete [10844.133414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10844.142382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10844.147322] Lustre: Skipped 1 previous similar message [10844.155291] Lustre: Skipped 1 previous similar message [10848.162472] Lustre: server umount lustre-MDT0000 complete [10849.254042] LustreError: 366140:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10849.299327] LustreError: 366140:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [10852.623799] LustreError: 366124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855003 with bad export cookie 7599470275321007736 [10852.635961] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10852.640858] LustreError: 366124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10853.130135] Lustre: server umount lustre-MDT0001 complete [10870.570632] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [10874.618835] Key type lgssc unregistered [10875.016940] LNet: 369680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10875.038879] LNetError: 369680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10875.056732] LNet: Removed LNI 192.168.203.106@tcp [10876.209544] Key type .llcrypt unregistered [10876.211969] Key type ._llcrypt unregistered [10906.917371] Key type ._llcrypt registered [10906.920830] Key type .llcrypt registered [10907.069533] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [10908.353761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10908.532521] alg: No test for adler32 (adler32-zlib) [10909.783402] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [10910.204691] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [10911.975178] Key type lgssc registered [10913.947813] Lustre: Echo OBD driver; http://www.lustre.org/ [10923.475711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10923.494613] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10923.503726] LustreError: 371123:0:(osd_handler.c:8550:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10923.531344] LustreError: 371123:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10923.544702] LustreError: 371123:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10923.561354] LustreError: 371123:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10923.573844] LustreError: 371123:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10931.890407] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10931.896229] LustreError: 371520:0:(osd_handler.c:8550:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10931.915645] LustreError: 371520:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10931.924925] LustreError: 371520:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10931.933341] LustreError: 371520:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10931.943379] LustreError: 371520:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10954.344946] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [10957.595581] Key type lgssc unregistered [10957.954800] LNet: 372529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10957.967121] LNetError: 372529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10958.003183] LNet: Removed LNI 192.168.203.106@tcp [10959.160468] Key type .llcrypt unregistered [10959.162252] Key type ._llcrypt unregistered [10980.418622] Key type ._llcrypt registered [10980.422402] Key type .llcrypt registered [10980.585624] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [10999.076723] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [11000.552303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11000.687653] alg: No test for adler32 (adler32-zlib) [11002.191559] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [11002.576248] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [11004.479194] Key type lgssc registered [11006.093300] Lustre: Echo OBD driver; http://www.lustre.org/ [11047.770955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11047.806414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11049.076708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11049.193662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11049.284861] Lustre: lustre-MDT0000: new disk, initializing [11049.404906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11049.446636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11055.382552] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11067.739745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11067.930445] Lustre: 375685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11068.018159] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11068.024182] Lustre: Skipped 1 previous similar message [11068.189689] Lustre: lustre-MDT0001: new disk, initializing [11068.375338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11068.475959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11068.497106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11073.809548] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11084.586262] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11090.660113] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11098.098668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11098.476530] Lustre: lustre-OST0000: new disk, initializing [11098.480168] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11098.486860] Lustre: 377654:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11098.638221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11105.709546] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11106.410453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11106.426561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11106.585985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11115.946790] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11121.508739] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11121.751754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11126.604826] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11126.841961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11132.386779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11132.390561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11132.402779] LustreError: Skipped 1 previous similar message [11132.424177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11132.425603] Lustre: Skipped 1 previous similar message [11134.980593] Lustre: server umount lustre-OST0000 complete [11140.064357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11140.085651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11140.114825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11140.118010] Lustre: Skipped 1 previous similar message [11142.635570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11142.649064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11147.754364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11147.758561] Lustre: Skipped 1 previous similar message [11151.905551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11151.912269] Lustre: Skipped 2 previous similar messages [11153.375278] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11153.578196] Lustre: server umount lustre-MDT0000 complete [11158.505112] LustreError: 375679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855309 with bad export cookie 3197133805536265302 [11158.515308] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11158.515686] LustreError: 375679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11159.159875] Lustre: server umount lustre-MDT0001 complete [11170.984642] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11172.953296] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 14:28:43 (1787855323) [11180.433182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11181.075453] LustreError: 380116:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11181.097531] LustreError: 380116:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [11181.189612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11184.971850] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11186.664795] LustreError: 380117:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11191.777741] LustreError: 380116:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11195.897507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11196.299506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11204.146961] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11214.151964] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11221.650325] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11229.491138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11229.738364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11236.967626] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11245.294362] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11253.030893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11253.199294] Lustre: lustre-OST0001: new disk, initializing [11253.203566] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11253.209560] Lustre: 382956:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11253.311569] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11259.879979] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11261.574862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11261.593399] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11261.755727] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11269.381341] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11277.286762] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11277.296195] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11277.328763] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11277.337973] Lustre: Skipped 1 previous similar message [11280.529595] Lustre: server umount lustre-OST0001 complete [11282.406862] LustreError: 381940:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11282.421705] LustreError: 381940:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [11287.528304] LustreError: 381939:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11287.547858] LustreError: 381939:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [11297.253699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11297.254018] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11297.259376] LustreError: Skipped 2 previous similar messages [11297.260299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11297.271822] Lustre: Skipped 2 previous similar messages [11297.292261] Lustre: Skipped 2 previous similar messages [11297.764481] LustreError: 381940:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11297.789512] LustreError: 381940:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [11303.264597] Lustre: server umount lustre-OST0000 complete [11308.003127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11308.033971] Lustre: Skipped 1 previous similar message [11313.840978] Lustre: server umount lustre-MDT0000 complete [11317.175897] LustreError: 382955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855468 with bad export cookie 3197133805536266184 [11317.178604] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11317.190056] LustreError: 382955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11323.780746] Lustre: server umount lustre-MDT0001 complete [11334.499845] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [11337.143594] Key type lgssc unregistered [11337.417758] LNet: 384947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11337.421887] LNetError: 384947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11337.431169] LNet: Removed LNI 192.168.203.106@tcp [11338.189889] Key type .llcrypt unregistered [11338.192467] Key type ._llcrypt unregistered [11386.684810] Key type ._llcrypt registered [11386.686874] Key type .llcrypt registered [11386.809352] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [11388.555534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11388.580853] alg: No test for adler32 (adler32-zlib) [11389.776710] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [11390.020548] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [11391.735160] Key type lgssc registered [11392.779552] Lustre: Echo OBD driver; http://www.lustre.org/ [11400.463221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11400.487995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11401.771899] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11401.808681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11402.009915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11406.700818] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11415.294602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11415.422586] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11415.461712] Lustre: 387745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11415.724259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11420.258178] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11427.436288] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11433.476962] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11436.513540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11436.524726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11436.539171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11440.108995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11440.110816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11441.583944] Lustre: server umount lustre-MDT0000 complete [11445.442955] LustreError: 387737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855596 with bad export cookie 3919309621340404751 [11445.451496] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11445.457033] LustreError: 387737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11445.790760] Lustre: server umount lustre-MDT0001 complete [11455.139231] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 14:33:25 (1787855605) [11460.808520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11461.238994] LustreError: 390257:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11461.254916] LustreError: 390257:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [11461.276872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11465.515598] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11466.729992] LustreError: 390258:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11471.855437] LustreError: 390257:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11473.892471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11474.377551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11478.410851] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11485.839620] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11491.298450] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11497.062945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11497.262624] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11497.391357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11502.731342] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11509.211730] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11516.616641] Lustre: Permanently deactivating lustre-OST0000 [11516.631660] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11524.131765] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11525.250210] LustreError: 393244:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11525.260029] LustreError: 393244:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11525.270450] LustreError: 393244:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11528.324492] LustreError: 393292:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11528.329592] LustreError: 393292:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11536.072540] Lustre: server umount lustre-OST0000 complete [11538.432610] LustreError: 393542:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11538.451896] LustreError: 393542:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11540.967285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11540.983662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11541.005919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11541.016301] Lustre: Skipped 1 previous similar message [11544.036993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11544.040087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11549.155589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11549.164038] Lustre: Skipped 2 previous similar messages [11550.328986] Lustre: server umount lustre-MDT0000 complete [11554.274183] LustreError: 390985:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11554.305325] LustreError: 390985:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [11554.823673] LustreError: 392092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855706 with bad export cookie 3919309621340405332 [11554.831291] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11554.843977] LustreError: 392092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11565.528828] Lustre: server umount lustre-MDT0001 complete [11576.342373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11581.506679] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11584.959031] LustreError: 394703:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11591.949153] Lustre: 395042:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.106@tcp [11594.201695] Lustre: server umount MGS complete [11605.862570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11606.292432] LustreError: 395766:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11606.320714] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11606.330695] Lustre: Skipped 1 previous similar message [11606.383630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11610.805294] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11616.738629] LustreError: 395766:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11616.771250] LustreError: 395766:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [11619.270242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11619.619328] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11623.949175] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11630.636332] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11635.438695] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11636.909948] Lustre: Permanently reactivating lustre-OST0000 [11636.934010] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11636.943279] Lustre: Skipped 2 previous similar messages [11666.400496] LustreError: 397596:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11673.368386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11673.664314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11673.671515] Lustre: Skipped 1 previous similar message [11680.465838] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11687.951415] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11694.984114] Lustre: server umount lustre-OST0000 complete [11701.728749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11701.741779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11701.752029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11709.483337] Lustre: server umount lustre-MDT0000 complete [11712.486726] LustreError: 395767:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11712.512783] LustreError: 395767:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [11712.573345] LustreError: 395747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855863 with bad export cookie 3919309621340406375 [11712.584301] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11712.585466] LustreError: 395747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11722.930986] Lustre: server umount lustre-MDT0001 complete [11734.085244] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [11737.005935] Key type lgssc unregistered [11737.271737] LNet: 400232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11737.282549] LNetError: 400232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11737.325552] LNet: Removed LNI 192.168.203.106@tcp [11738.184688] Key type .llcrypt unregistered [11738.191246] Key type ._llcrypt unregistered [11756.032277] Key type ._llcrypt registered [11756.033948] Key type .llcrypt registered [11756.165280] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [11768.640667] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [11769.725977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11769.745973] alg: No test for adler32 (adler32-zlib) [11770.784896] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [11770.988541] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [11772.679162] Key type lgssc registered [11773.535521] Lustre: Echo OBD driver; http://www.lustre.org/ [11809.660108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11809.695582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11811.189857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11811.276462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11811.448220] Lustre: lustre-MDT0000: new disk, initializing [11811.661037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11811.714543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11816.486368] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11827.172854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11827.319290] Lustre: 403364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11827.374340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11827.380599] Lustre: Skipped 1 previous similar message [11827.442610] Lustre: lustre-MDT0001: new disk, initializing [11827.536875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11827.580057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11827.610977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11832.741851] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11842.867856] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11848.404407] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11855.747320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11856.208914] Lustre: lustre-OST0000: new disk, initializing [11856.214658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11856.221708] Lustre: 405333:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11856.344220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11862.990746] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11863.624262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11863.637341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11863.752547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11871.907772] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11877.011323] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11877.215586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11882.131796] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11882.302908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11884.512364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11884.514855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11884.518777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11884.523966] Lustre: Skipped 1 previous similar message [11889.645472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11889.656791] Lustre: Skipped 2 previous similar messages [11894.754786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11894.763792] Lustre: Skipped 1 previous similar message [11898.335127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11898.538165] Lustre: server umount lustre-OST0000 complete [11904.485948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11904.496537] LustreError: Skipped 1 previous similar message [11904.502682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11904.515559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11910.119949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11910.122920] Lustre: Skipped 2 previous similar messages [11916.255258] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11916.498334] Lustre: server umount lustre-MDT0000 complete [11920.508029] LustreError: 403357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856071 with bad export cookie 16075182565121938955 [11920.510810] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11920.518518] LustreError: 403357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11920.871932] Lustre: server umount lustre-MDT0001 complete [11929.166908] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11930.647535] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 14:41:21 (1787856081) [11937.364422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11937.939109] LustreError: 407798:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11937.960500] LustreError: 407798:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [11938.014442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11942.280376] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11943.398227] LustreError: 407799:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11948.530473] LustreError: 407798:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11950.517980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11950.795457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11954.869970] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11962.714862] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11968.659886] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11975.638441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11975.967062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11982.300973] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11990.717663] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11992.712450] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11995.697836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12006.135771] Lustre: server umount lustre-OST0000 complete [12010.978903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12010.998686] Lustre: Skipped 1 previous similar message [12011.022774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12011.044282] Lustre: Skipped 3 previous similar messages [12024.287162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12024.512243] Lustre: server umount lustre-MDT0000 complete [12026.339369] LustreError: 407799:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12026.377717] LustreError: 407799:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [12027.368599] Lustre: 401591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856162/real 1787856162] req@00000000cd0a3beb x1874702811339392/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787856178 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12027.409345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12027.426853] Lustre: Skipped 1 previous similar message [12028.435183] LustreError: 407778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856179 with bad export cookie 16075182565121939830 [12028.456186] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12028.459210] LustreError: 407778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12028.807381] Lustre: server umount lustre-MDT0001 complete [12039.714467] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [12042.198278] Key type lgssc unregistered [12042.485277] LNet: 411526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12042.496804] LNetError: 411526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12042.514332] LNet: Removed LNI 192.168.203.106@tcp [12043.312859] Key type .llcrypt unregistered [12043.315954] Key type ._llcrypt unregistered [12064.981226] Key type ._llcrypt registered [12064.985897] Key type .llcrypt registered [12065.102870] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [12065.385163] Key type .llcrypt unregistered [12065.387034] Key type ._llcrypt unregistered [12081.635501] Key type ._llcrypt registered [12081.637923] Key type .llcrypt registered [12081.724827] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [12082.620196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12082.735441] alg: No test for adler32 (adler32-zlib) [12083.870119] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [12084.224506] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [12085.903177] Key type lgssc registered [12086.875774] Lustre: Echo OBD driver; http://www.lustre.org/ [12094.133379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12094.155343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12095.544025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12099.708399] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12105.706941] LustreError: 413744:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12105.724027] LustreError: 413744:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [12110.568406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12110.818113] LustreError: 413743:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12111.144422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12116.453732] LustreError: 413744:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12116.502811] LustreError: 413744:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [12120.547349] LustreError: 413743:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12120.580626] LustreError: 413743:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [12120.592332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12121.612617] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12125.666755] LustreError: 413744:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12125.697163] LustreError: 413744:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [12131.425605] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12135.917334] LustreError: 413737:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.932641] LustreError: 413737:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [12138.739724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12139.107808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12143.696537] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12154.337053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12154.345925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12154.361762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12154.849587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12154.869249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12158.934095] Lustre: server umount lustre-OST0000 complete [12164.576152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12164.580710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12164.596790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12169.097562] Lustre: server umount lustre-MDT0000 complete [12170.210518] LustreError: 413743:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12170.232070] LustreError: 413743:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [12172.548239] LustreError: 415851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856323 with bad export cookie 14731938434847137179 [12172.558370] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12172.559267] LustreError: 415851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12172.863813] Lustre: server umount lustre-MDT0001 complete [12184.068305] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [12188.062097] Key type lgssc unregistered [12188.399327] LNet: 416895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12188.405378] LNetError: 416895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12189.485042] LNet: Removed LNI 192.168.203.106@tcp [12190.546911] Key type .llcrypt unregistered [12190.549952] Key type ._llcrypt unregistered [12215.759718] Key type ._llcrypt registered [12215.763508] Key type .llcrypt registered [12215.874847] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [12216.873961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12217.016904] alg: No test for adler32 (adler32-zlib) [12218.157055] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [12218.352712] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [12220.047183] Key type lgssc registered [12221.208735] Lustre: Echo OBD driver; http://www.lustre.org/ [12228.677727] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12228.708332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12244.384797] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12249.503679] LustreError: 418289:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [12276.255122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12283.080948] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12290.771105] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12298.223466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12298.703993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12298.753196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12303.351569] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12303.856830] LustreError: 418314:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12303.890145] LustreError: 418314:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [12308.964655] LustreError: 418315:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12311.010651] LustreError: 418314:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12311.025021] LustreError: 418314:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [12311.708397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12312.087358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12316.719314] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12324.248466] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12330.490466] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12339.680909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12339.699859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12339.725651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12340.143290] Lustre: server umount lustre-OST0000 complete [12345.832885] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12345.841851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12345.855381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12346.555545] Lustre: server umount lustre-MDT0000 complete [12347.873590] LustreError: 419332:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12347.874184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12349.680926] LustreError: 418296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856500 with bad export cookie 15235445275757026930 [12349.682702] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12349.690188] LustreError: 418296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12359.319367] Lustre: server umount lustre-MDT0001 complete [12373.200105] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [12375.721314] Key type lgssc unregistered [12375.956257] LNet: 421866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12375.969755] LNetError: 421866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12375.987109] LNet: Removed LNI 192.168.203.106@tcp [12376.715460] Key type .llcrypt unregistered [12376.719083] Key type ._llcrypt unregistered [12401.891715] Key type ._llcrypt registered [12401.896524] Key type .llcrypt registered [12401.975691] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [12403.336311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12403.380893] alg: No test for adler32 (adler32-zlib) [12404.515424] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [12404.905819] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [12406.615882] Key type lgssc registered [12408.046859] Lustre: Echo OBD driver; http://www.lustre.org/ [12416.768419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12416.797155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12418.366092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12422.389690] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12428.770109] LustreError: 423284:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12428.789304] LustreError: 423284:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [12430.679411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12430.979974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12434.687446] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12442.078850] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12448.293367] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12455.175890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12455.592452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12462.309178] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12463.804108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12470.578466] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12476.899945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12476.913437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12476.928866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12477.542473] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [12479.462706] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12479.465227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12479.484424] Lustre: Skipped 1 previous similar message [12479.492222] Lustre: Skipped 1 previous similar message [12480.703113] Lustre: server umount lustre-MDT0000 complete [12483.132192] LustreError: 425127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.6@tcp arrived at 1787856634 with bad export cookie 12210545089346933011 [12484.577699] LustreError: 423281:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12484.597214] LustreError: 423281:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [12484.608147] LustreError: 423264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856635 with bad export cookie 12210545089346932115 [12484.619197] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12484.623374] LustreError: 423264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12487.648199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12487.654759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12487.672865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12487.681074] Lustre: Skipped 1 previous similar message [12489.698217] LustreError: 424011:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12489.714217] LustreError: 424011:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [12490.803497] Lustre: server umount lustre-OST0000 complete [12494.820731] LustreError: 423284:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12497.200983] Lustre: server umount lustre-MDT0001 complete [12507.882323] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [12510.532844] Key type lgssc unregistered [12510.841454] LNet: 426970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12510.847248] LNetError: 426970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12510.861966] LNet: Removed LNI 192.168.203.106@tcp [12511.744548] Key type .llcrypt unregistered [12511.747792] Key type ._llcrypt unregistered [12535.761872] Key type ._llcrypt registered [12535.764703] Key type .llcrypt registered [12535.902185] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [12537.021670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12537.056197] alg: No test for adler32 (adler32-zlib) [12538.082897] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [12538.288557] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [12539.927195] Key type lgssc registered [12541.365974] Lustre: Echo OBD driver; http://www.lustre.org/ [12548.837634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12548.876277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12550.347238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12555.200925] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12559.845240] LustreError: 428389:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12559.871318] LustreError: 428389:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [12563.378610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12563.748715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12568.514525] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12575.590522] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12581.597333] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12588.501528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12589.052375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12595.205147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12595.268461] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12603.239477] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12610.016853] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12610.028407] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12610.046978] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12610.535678] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12610.559868] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12610.569913] Lustre: Skipped 1 previous similar message [12613.818306] Lustre: server umount lustre-MDT0001 complete [12615.235670] LustreError: 428383:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12620.768313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12620.769188] LustreError: 428390:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12620.780442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12620.783403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12620.820408] LustreError: 428390:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [12625.893134] LustreError: 428389:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12625.895239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12626.202851] Lustre: server umount lustre-OST0000 complete [12630.772163] Lustre: server umount lustre-MDT0000 complete [12647.947760] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [12650.757124] Key type lgssc unregistered [12650.943762] LNet: 432077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12650.948667] LNetError: 432077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12650.969251] LNet: Removed LNI 192.168.203.106@tcp [12651.811454] Key type .llcrypt unregistered [12651.816867] Key type ._llcrypt unregistered [12679.562478] Key type ._llcrypt registered [12679.565486] Key type .llcrypt registered [12679.656621] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [12695.782458] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [12697.001838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12697.032188] alg: No test for adler32 (adler32-zlib) [12698.192517] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [12698.588464] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [12700.375796] Key type lgssc registered [12701.444481] Lustre: Echo OBD driver; http://www.lustre.org/ [12739.643326] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [12754.712224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12754.755048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12755.982657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12756.002673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12756.087395] Lustre: lustre-MDT0000: new disk, initializing [12756.143501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12756.182180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12760.811970] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12776.390069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12776.590268] Lustre: lustre-OST0000: new disk, initializing [12776.594363] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12776.599485] Lustre: Skipped 1 previous similar message [12776.604746] Lustre: 437111:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12776.679051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12784.816548] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12785.856513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12785.874752] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12785.919048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12797.796268] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12815.229148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12815.413560] Lustre: 436094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12815.479554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12815.583760] Lustre: lustre-MDT0001: new disk, initializing [12815.732208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12815.786236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12815.796112] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12819.859352] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12828.546891] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12828.753730] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12833.624704] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12833.878544] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12842.091547] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.6@tcp (stopping) [12844.512616] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12844.539793] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12844.554749] Lustre: Skipped 1 previous similar message [12844.554813] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12844.580734] Lustre: Skipped 1 previous similar message [12845.733648] Lustre: server umount lustre-MDT0001 complete [12854.767761] LustreError: 436106:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12854.791313] LustreError: 436106:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 5 previous similar messages [12859.874422] LustreError: 436100:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12859.902173] LustreError: 436100:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [12864.997290] LustreError: 436106:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12865.001846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12865.027856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12870.118911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12870.121051] LustreError: 436105:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12875.232897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12875.234227] LustreError: 436106:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12878.304068] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12878.497499] Lustre: server umount lustre-OST0000 complete [12896.223234] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12896.588322] Lustre: server umount lustre-MDT0000 complete [12909.966789] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [12912.444648] Key type lgssc unregistered [12912.740854] LNet: 440864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12912.748040] LNetError: 440864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12912.792134] LNet: Removed LNI 192.168.203.106@tcp [12913.786343] Key type .llcrypt unregistered [12913.789094] Key type ._llcrypt unregistered [12954.558649] Key type ._llcrypt registered [12954.561663] Key type .llcrypt registered [12954.676752] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [12954.970509] Key type .llcrypt unregistered [12954.972208] Key type ._llcrypt unregistered [12971.338504] Key type ._llcrypt registered [12971.343572] Key type .llcrypt registered [12971.481593] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [12972.421354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12972.482728] alg: No test for adler32 (adler32-zlib) [12973.589701] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [12973.891962] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [12975.618455] Key type lgssc registered [12977.020040] Lustre: Echo OBD driver; http://www.lustre.org/ [13005.218222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13005.279504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13006.715570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13006.876052] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13007.051100] Lustre: lustre-MDT0000: new disk, initializing [13007.184322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13007.229326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13013.304716] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13024.919252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13025.126954] Lustre: 444245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13025.226381] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13025.231616] Lustre: Skipped 1 previous similar message [13025.366635] Lustre: lustre-MDT0001: new disk, initializing [13025.431875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13025.459918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13025.468042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13029.786377] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13040.921689] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13048.413718] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13055.878076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13056.236435] Lustre: lustre-OST0000: new disk, initializing [13056.243721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13056.255995] Lustre: 446218:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13056.374690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13063.097953] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13066.286231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13066.293747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13066.420218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13072.682085] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13082.085540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13082.090712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13082.090845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13082.107614] Lustre: Skipped 1 previous similar message [13086.002131] Lustre: server umount lustre-MDT0000 complete [13089.531584] LustreError: 446217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857240 with bad export cookie 11296274326088799011 [13089.537755] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13089.540891] LustreError: 446217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13089.865952] Lustre: server umount lustre-MDT0001 complete [13103.711903] Lustre: server umount lustre-OST0000 complete [13113.867286] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:01:04 (1787857264) [13121.078825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13121.605224] LustreError: 448254:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13121.627473] LustreError: 448254:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 4 previous similar messages [13121.675402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13126.449991] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13127.137366] LustreError: 448255:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13131.241296] LustreError: 448254:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13135.170855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13135.585274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13139.920395] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13147.408814] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13152.194764] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13172.517050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13172.713279] Lustre: Found index 0 for lustre-OST0000, updating log [13172.870315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13178.865198] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13186.939716] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13192.369228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13197.803445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13197.807254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13197.807377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13197.807381] Lustre: Skipped 2 previous similar messages [13197.815099] Lustre: Skipped 1 previous similar message [13198.884141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13198.899648] Lustre: Skipped 2 previous similar messages [13199.179516] Lustre: server umount lustre-OST0000 complete [13215.201234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13215.222748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13215.234214] Lustre: Skipped 1 previous similar message [13220.321123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13220.324766] Lustre: Skipped 1 previous similar message [13221.038517] Lustre: server umount lustre-MDT0000 complete [13224.398624] LustreError: 448235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857375 with bad export cookie 11296274326088800404 [13224.404265] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13224.409519] LustreError: 448235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13224.714176] Lustre: server umount lustre-MDT0001 complete [13234.060509] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:03:04 (1787857384) [13236.063771] LNet: 1 local NIs in recovery (showing 1): 192.168.203.106@tcp [13854.826668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13855.382598] LustreError: 454664:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13855.480371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13860.834309] LustreError: 454665:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13860.895838] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13864.931505] LustreError: 454664:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13870.192429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13870.512954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13874.648964] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13882.774849] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13889.901760] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13911.419193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13911.682296] Lustre: Found index 0 for lustre-OST0000, updating log [13912.075913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13916.000031] LNet: 1 local NIs in recovery (showing 1): 192.168.203.106@tcp [13919.846854] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13929.007752] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13932.260170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13932.268172] Lustre: Skipped 1 previous similar message [13932.273144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13937.561120] Lustre: server umount lustre-OST0000 complete [13941.145959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13941.161704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13943.206153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13943.212915] Lustre: Skipped 4 previous similar messages [13947.299386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13947.306637] Lustre: Skipped 7 previous similar messages [13947.465139] Lustre: server umount lustre-MDT0000 complete [13948.331757] LustreError: 454664:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13948.361572] LustreError: 454664:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [13951.430643] LustreError: 454646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858102 with bad export cookie 11296274326088801489 [13951.440192] LustreError: 454646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13951.445681] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13951.836276] Lustre: server umount lustre-MDT0001 complete [13976.417115] LNet: 1 local NIs in recovery (showing 1): 192.168.203.106@tcp [13981.819537] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [13993.278915] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [14036.035976] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 15:16:25 (1787858185) [14054.436982] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [14057.252635] Key type lgssc unregistered [14057.512731] LNet: 461288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14057.521147] LNetError: 461288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14173.288687] LNet: Removed LNI 192.168.203.106@tcp [14174.120024] Key type .llcrypt unregistered [14174.122988] Key type ._llcrypt unregistered [14190.350485] Key type ._llcrypt registered [14190.354396] Key type .llcrypt registered [14190.436688] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [14191.493956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14191.534027] alg: No test for adler32 (adler32-zlib) [14192.616193] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [14192.867083] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [14194.591232] Key type lgssc registered [14195.800813] Lustre: Echo OBD driver; http://www.lustre.org/ [14199.844236] LNet: Added LNI 192.168.203.106@tcp100 [8/256/0/180] [14200.630194] LNet: Added LNI 192.168.203.106@tcp101 [8/256/0/180] [14201.438130] LNet: Added LNI 192.168.203.106@tcp102 [8/256/0/180] [14202.186614] LNet: Added LNI 192.168.203.106@tcp103 [8/256/0/180] [14203.190742] LNet: Added LNI 192.168.203.106@tcp104 [8/256/0/180] [14203.971932] LNet: Added LNI 192.168.203.106@tcp105 [8/256/0/180] [14204.928785] LNet: Added LNI 192.168.203.106@tcp106 [8/256/0/180] [14205.625185] LNet: Added LNI 192.168.203.106@tcp107 [8/256/0/180] [14206.460460] LNet: Added LNI 192.168.203.106@tcp108 [8/256/0/180] [14207.212154] LNet: Added LNI 192.168.203.106@tcp109 [8/256/0/180] [14208.031864] LNet: Added LNI 192.168.203.106@tcp110 [8/256/0/180] [14208.937816] LNet: Added LNI 192.168.203.106@tcp111 [8/256/0/180] [14209.708446] LNet: Added LNI 192.168.203.106@tcp112 [8/256/0/180] [14210.403712] LNet: Added LNI 192.168.203.106@tcp113 [8/256/0/180] [14211.097135] LNet: Added LNI 192.168.203.106@tcp114 [8/256/0/180] [14211.857234] LNet: Added LNI 192.168.203.106@tcp115 [8/256/0/180] [14212.593967] LNet: Added LNI 192.168.203.106@tcp116 [8/256/0/180] [14213.576145] LNet: Added LNI 192.168.203.106@tcp117 [8/256/0/180] [14214.382715] LNet: Added LNI 192.168.203.106@tcp118 [8/256/0/180] [14215.371650] LNet: Added LNI 192.168.203.106@tcp119 [8/256/0/180] [14216.193938] LNet: Added LNI 192.168.203.106@tcp120 [8/256/0/180] [14217.095398] LNet: Added LNI 192.168.203.106@tcp121 [8/256/0/180] [14217.829367] LNet: Added LNI 192.168.203.106@tcp122 [8/256/0/180] [14218.919373] LNet: Added LNI 192.168.203.106@tcp123 [8/256/0/180] [14220.122765] LNet: Added LNI 192.168.203.106@tcp124 [8/256/0/180] [14221.157198] LNet: Added LNI 192.168.203.106@tcp125 [8/256/0/180] [14222.405153] LNet: Added LNI 192.168.203.106@tcp126 [8/256/0/180] [14223.565551] LNet: Added LNI 192.168.203.106@tcp127 [8/256/0/180] [14224.736652] LNet: Added LNI 192.168.203.106@tcp128 [8/256/0/180] [14226.098029] LNet: Added LNI 192.168.203.106@tcp129 [8/256/0/180] [14227.122815] LNet: Added LNI 192.168.203.106@tcp130 [8/256/0/180] [14228.059558] LNet: Added LNI 192.168.203.106@tcp131 [8/256/0/180] [14229.078454] LNet: Added LNI 192.168.203.106@tcp132 [8/256/0/180] [14229.988814] LNet: Added LNI 192.168.203.106@tcp133 [8/256/0/180] [14230.987686] LNet: Added LNI 192.168.203.106@tcp134 [8/256/0/180] [14232.090746] LNet: Added LNI 192.168.203.106@tcp135 [8/256/0/180] [14238.894069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14238.943390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14240.026456] Lustre: 464322:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.106@tcp: can't add NID 192.168.203.106@tcp130: rc = -75 [14240.206430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14240.243285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14240.389600] Lustre: lustre-MDT0000: new disk, initializing [14240.517375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14240.549501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14245.468632] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14255.682654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14255.798303] Lustre: 464340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14255.870350] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14255.874707] Lustre: Skipped 1 previous similar message [14255.970779] Lustre: lustre-MDT0001: new disk, initializing [14255.994135] Lustre: 465135:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.106@tcp: can't add NID 192.168.203.106@tcp130: rc = -75 [14256.001501] Lustre: 465135:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14256.128070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14256.180367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14256.194537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14260.730603] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14269.800112] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14275.660996] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14282.041741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14282.336336] Lustre: lustre-OST0000: new disk, initializing [14282.341247] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14282.356978] Lustre: 466311:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14282.528216] Lustre: 466316:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.106@tcp: can't add NID 192.168.203.106@tcp130: rc = -75 [14282.545474] Lustre: 466316:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14282.620445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14288.595272] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14292.528852] Lustre: 466828:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.106@tcp: can't add NID 192.168.203.106@tcp130: rc = -75 [14292.538133] Lustre: 466828:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14292.609735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14292.622667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14292.740133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14297.647600] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14308.333324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14308.334829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14308.339881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14308.343896] Lustre: Skipped 1 previous similar message [14308.349724] Lustre: Skipped 1 previous similar message [14311.627546] Lustre: server umount lustre-OST0000 complete [14317.537689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14317.547518] LustreError: Skipped 1 previous similar message [14317.559119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14317.578327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14318.575034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14318.592855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14318.604833] Lustre: Skipped 1 previous similar message [14323.682861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14323.698440] Lustre: Skipped 1 previous similar message [14328.800734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14328.805381] Lustre: Skipped 1 previous similar message [14329.311168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14329.577683] Lustre: server umount lustre-MDT0000 complete [14333.445843] LustreError: 464330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858484 with bad export cookie 3166151397729859665 [14333.462933] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14333.468056] LustreError: 464330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14333.880451] Lustre: server umount lustre-MDT0001 complete [14346.458391] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [14349.644295] Key type lgssc unregistered [14350.245020] LNet: 468226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14350.252921] LNetError: 468226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14351.327988] LNet: Removed LNI 192.168.203.106@tcp [14351.332967] LNetError: 468226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14351.341148] LNet: Removed LNI 192.168.203.106@tcp100 [14352.415156] LNet: Removed LNI 192.168.203.106@tcp101 [14352.419531] LNetError: 468226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14352.435323] LNetError: 468226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14352.452925] LNet: Removed LNI 192.168.203.106@tcp102 [14352.460305] LNet: Removed LNI 192.168.203.106@tcp103 [14352.469219] LNet: Removed LNI 192.168.203.106@tcp104 [14352.478207] LNet: Removed LNI 192.168.203.106@tcp105 [14352.481742] LNet: Removed LNI 192.168.203.106@tcp106 [14352.485663] LNet: Removed LNI 192.168.203.106@tcp107 [14352.498127] LNet: Removed LNI 192.168.203.106@tcp108 [14352.508697] LNet: Removed LNI 192.168.203.106@tcp109 [14352.512086] LNet: Removed LNI 192.168.203.106@tcp110 [14352.528607] LNet: Removed LNI 192.168.203.106@tcp111 [14352.534093] LNet: Removed LNI 192.168.203.106@tcp112 [14352.540307] LNet: Removed LNI 192.168.203.106@tcp113 [14352.545284] LNet: Removed LNI 192.168.203.106@tcp114 [14352.550716] LNet: Removed LNI 192.168.203.106@tcp115 [14352.554830] LNet: Removed LNI 192.168.203.106@tcp116 [14352.559876] LNet: Removed LNI 192.168.203.106@tcp117 [14352.565616] LNet: Removed LNI 192.168.203.106@tcp118 [14352.572805] LNet: Removed LNI 192.168.203.106@tcp119 [14352.577522] LNet: Removed LNI 192.168.203.106@tcp120 [14352.580975] LNet: Removed LNI 192.168.203.106@tcp121 [14352.587653] LNet: Removed LNI 192.168.203.106@tcp122 [14352.592627] LNet: Removed LNI 192.168.203.106@tcp123 [14352.596458] LNet: Removed LNI 192.168.203.106@tcp124 [14352.600354] LNet: Removed LNI 192.168.203.106@tcp125 [14352.602845] LNet: Removed LNI 192.168.203.106@tcp126 [14352.612797] LNet: Removed LNI 192.168.203.106@tcp127 [14352.619754] LNet: Removed LNI 192.168.203.106@tcp128 [14352.624459] LNet: Removed LNI 192.168.203.106@tcp129 [14352.627524] LNet: Removed LNI 192.168.203.106@tcp130 [14352.638541] LNet: Removed LNI 192.168.203.106@tcp131 [14352.642277] LNet: Removed LNI 192.168.203.106@tcp132 [14352.644488] LNet: Removed LNI 192.168.203.106@tcp133 [14352.651094] LNet: Removed LNI 192.168.203.106@tcp134 [14352.665557] LNet: Removed LNI 192.168.203.106@tcp135 [14353.488216] Key type .llcrypt unregistered [14353.489769] Key type ._llcrypt unregistered [14373.325354] Key type ._llcrypt registered [14373.328145] Key type .llcrypt registered [14373.418543] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [14386.097903] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [14387.029181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14387.189083] alg: No test for adler32 (adler32-zlib) [14388.269436] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [14388.538676] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [14390.176118] Key type lgssc registered [14391.196858] Lustre: Echo OBD driver; http://www.lustre.org/ [14425.121572] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:22:55 (1787858575) [14444.306454] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [14447.344721] Key type lgssc unregistered [14447.712934] LNet: 472293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14447.725252] LNetError: 472293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14447.751517] LNet: Removed LNI 192.168.203.106@tcp [14448.758973] Key type .llcrypt unregistered [14448.764456] Key type ._llcrypt unregistered [14464.768987] Key type ._llcrypt registered [14464.770913] Key type .llcrypt registered [14464.862337] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [14465.773839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14465.921038] alg: No test for adler32 (adler32-zlib) [14466.993231] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [14467.230270] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [14468.919166] Key type lgssc registered [14469.959476] Lustre: Echo OBD driver; http://www.lustre.org/ [14478.953289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14478.978937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14480.300111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14480.356506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14480.441269] Lustre: lustre-MDT0000: new disk, initializing [14480.589794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14480.615674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14486.476932] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14498.529433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14498.698495] Lustre: 473507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14498.763789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14498.769346] Lustre: Skipped 1 previous similar message [14498.895760] Lustre: lustre-MDT0001: new disk, initializing [14499.034338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14499.094853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14499.105061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14504.028063] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14513.020554] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14518.963086] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14525.290673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14525.628071] Lustre: lustre-OST0000: new disk, initializing [14525.631704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14525.637710] Lustre: 475475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14525.744149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14531.812406] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14533.716064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14533.732602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14533.822483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14540.364687] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14543.955239] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing lnet_if_list [14549.735696] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14549.791754] LNet: Added LNI 192.168.203.106@tcp10 [8/256/0/180] [14557.802641] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14557.829187] LNet: Added LNI 192.168.203.106@tcp11 [8/256/0/180] [14563.560511] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14563.594459] LNet: Added LNI 192.168.203.106@tcp12 [8/256/0/180] [14568.751652] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14568.780547] LNet: Added LNI 192.168.203.106@tcp13 [8/256/0/180] [14574.192292] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14574.214615] LNet: Added LNI 192.168.203.106@tcp14 [8/256/0/180] [14579.172169] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14579.211913] LNet: Added LNI 192.168.203.106@tcp15 [8/256/0/180] [14584.273664] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14584.333337] LNet: Added LNI 192.168.203.106@tcp16 [8/256/0/180] [14590.203530] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14590.232839] LNet: Added LNI 192.168.203.106@tcp17 [8/256/0/180] [14595.548503] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14595.589490] LNet: Added LNI 192.168.203.106@tcp18 [8/256/0/180] [14600.404258] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14600.434385] LNet: Added LNI 192.168.203.106@tcp19 [8/256/0/180] [14605.346950] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14605.381818] LNet: Added LNI 192.168.203.106@tcp20 [8/256/0/180] [14610.218744] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14610.245736] LNet: Added LNI 192.168.203.106@tcp21 [8/256/0/180] [14615.242867] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14615.288920] LNet: Added LNI 192.168.203.106@tcp22 [8/256/0/180] [14620.819985] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14620.846384] LNet: Added LNI 192.168.203.106@tcp23 [8/256/0/180] [14626.073788] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14626.102734] LNet: Added LNI 192.168.203.106@tcp24 [8/256/0/180] [14631.638901] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14631.678785] LNet: Added LNI 192.168.203.106@tcp25 [8/256/0/180] [14637.399792] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14637.424266] LNet: Added LNI 192.168.203.106@tcp26 [8/256/0/180] [14643.417273] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14643.459397] LNet: Added LNI 192.168.203.106@tcp27 [8/256/0/180] [14649.201990] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14649.248638] LNet: Added LNI 192.168.203.106@tcp28 [8/256/0/180] [14655.317783] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14655.359809] LNet: Added LNI 192.168.203.106@tcp29 [8/256/0/180] [14661.653747] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14661.690191] LNet: Added LNI 192.168.203.106@tcp30 [8/256/0/180] [14667.613657] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14667.641713] LNet: Added LNI 192.168.203.106@tcp31 [8/256/0/180] [14672.907867] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14672.928504] LNet: Added LNI 192.168.203.106@tcp32 [8/256/0/180] [14677.929231] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14677.948435] LNet: Added LNI 192.168.203.106@tcp33 [8/256/0/180] [14683.146392] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14683.211794] LNet: Added LNI 192.168.203.106@tcp34 [8/256/0/180] [14688.715060] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14688.743744] LNet: Added LNI 192.168.203.106@tcp35 [8/256/0/180] [14696.049257] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14696.074989] LNet: Added LNI 192.168.203.106@tcp36 [8/256/0/180] [14703.841198] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14703.880343] LNet: Added LNI 192.168.203.106@tcp37 [8/256/0/180] [14710.570717] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14710.611745] LNet: Added LNI 192.168.203.106@tcp38 [8/256/0/180] [14717.904973] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14717.931680] LNet: Added LNI 192.168.203.106@tcp39 [8/256/0/180] [14725.240638] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14725.272269] LNet: Added LNI 192.168.203.106@tcp40 [8/256/0/180] [14732.647245] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14732.672857] LNet: Added LNI 192.168.203.106@tcp41 [8/256/0/180] [14739.860695] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14739.915510] LNet: Added LNI 192.168.203.106@tcp42 [8/256/0/180] [14747.087656] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14747.108419] LNet: Added LNI 192.168.203.106@tcp43 [8/256/0/180] [14754.800303] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14754.839532] LNet: Added LNI 192.168.203.106@tcp44 [8/256/0/180] [14763.291249] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14763.332493] LNet: Added LNI 192.168.203.106@tcp45 [8/256/0/180] [14770.280603] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14770.303573] LNet: Added LNI 192.168.203.106@tcp46 [8/256/0/180] [14779.009680] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14779.071712] LNet: Added LNI 192.168.203.106@tcp47 [8/256/0/180] [14788.789773] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14788.822884] LNet: Added LNI 192.168.203.106@tcp48 [8/256/0/180] [14796.990075] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14797.015391] LNet: Added LNI 192.168.203.106@tcp49 [8/256/0/180] [14804.586332] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14804.615769] LNet: Added LNI 192.168.203.106@tcp50 [8/256/0/180] [14821.856599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14821.868678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14821.888196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14825.957427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14825.958122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14825.984119] Lustre: Skipped 1 previous similar message [14831.076048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14831.087400] Lustre: Skipped 4 previous similar messages [14836.191151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14836.192681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14836.216389] Lustre: Skipped 2 previous similar messages [14836.406545] Lustre: server umount lustre-MDT0000 complete [14839.567704] LustreError: 479257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858990 with bad export cookie 5217134933411616709 [14839.573429] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14839.579949] LustreError: 479257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14839.867892] Lustre: server umount lustre-MDT0001 complete [14856.855203] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [14857.695094] Lustre: 472779:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858992/real 1787858992] req@0000000037c2c263 x1874705638393728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787859008 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14857.708896] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14860.767337] Lustre: 472780:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858995/real 1787858995] req@00000000564d9b61 x1874705638393856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787859011 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14862.816399] Lustre: 472781:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858997/real 1787858997] req@00000000056cf0de x1874705638393984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787859013 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14864.579896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14864.927289] Lustre: 472778:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859000/real 1787859000] req@000000002843150a x1874705638394240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787859016 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14869.983318] Lustre: 472779:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859005/real 1787859005] req@000000005c6b4a05 x1874705638394368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787859021 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14884.321909] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4866fc949347e1ea [14884.339737] Lustre: MGC192.168.203.106@tcp: Connection restored to 0@lo (at 0@lo) [14884.761890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14888.339311] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14890.851930] LustreError: 475470:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14890.855772] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14890.867076] LustreError: 475470:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [14896.097610] LustreError: 475469:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14896.510361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14896.843711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14900.654980] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14901.934569] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14907.374095] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14914.032294] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14932.966818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14932.970440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14932.980568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14933.015323] Lustre: Skipped 1 previous similar message [14938.573769] Lustre: server umount lustre-OST0000 complete [14943.200169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14943.201690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14943.215803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14943.227767] Lustre: Skipped 4 previous similar messages [14948.064298] Lustre: server umount lustre-MDT0000 complete [14948.329022] LustreError: 499714:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14948.342096] LustreError: 499714:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [14951.690776] LustreError: 473500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859102 with bad export cookie 5217134933411619306 [14951.691916] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14951.701138] LustreError: 473500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14952.085087] Lustre: server umount lustre-MDT0001 complete [14963.110777] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [14965.710157] Key type lgssc unregistered [14966.102578] LNet: 502498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14966.122674] LNetError: 502498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14966.136466] LNet: Removed LNI 192.168.203.106@tcp [14966.139208] LNet: Removed LNI 192.168.203.106@tcp10 [14966.142532] LNet: Removed LNI 192.168.203.106@tcp11 [14966.152404] LNet: Removed LNI 192.168.203.106@tcp12 [14966.159450] LNet: Removed LNI 192.168.203.106@tcp13 [14966.165866] LNet: Removed LNI 192.168.203.106@tcp14 [14966.181261] LNet: Removed LNI 192.168.203.106@tcp15 [14966.188755] LNet: Removed LNI 192.168.203.106@tcp16 [14966.197300] LNet: Removed LNI 192.168.203.106@tcp17 [14966.200377] LNet: Removed LNI 192.168.203.106@tcp18 [14966.207846] LNet: Removed LNI 192.168.203.106@tcp19 [14966.217371] LNet: Removed LNI 192.168.203.106@tcp20 [14966.225589] LNet: Removed LNI 192.168.203.106@tcp21 [14966.230627] LNet: Removed LNI 192.168.203.106@tcp22 [14966.233335] LNet: Removed LNI 192.168.203.106@tcp23 [14966.239393] LNet: Removed LNI 192.168.203.106@tcp24 [14966.242428] LNet: Removed LNI 192.168.203.106@tcp25 [14966.245189] LNet: Removed LNI 192.168.203.106@tcp26 [14966.253821] LNet: Removed LNI 192.168.203.106@tcp27 [14966.255758] LNet: Removed LNI 192.168.203.106@tcp28 [14966.271252] LNet: Removed LNI 192.168.203.106@tcp29 [14966.274310] LNet: Removed LNI 192.168.203.106@tcp30 [14966.277391] LNet: Removed LNI 192.168.203.106@tcp31 [14966.282545] LNet: Removed LNI 192.168.203.106@tcp32 [14966.287901] LNet: Removed LNI 192.168.203.106@tcp33 [14966.290504] LNet: Removed LNI 192.168.203.106@tcp34 [14966.293767] LNet: Removed LNI 192.168.203.106@tcp35 [14966.296448] LNet: Removed LNI 192.168.203.106@tcp36 [14966.298764] LNet: Removed LNI 192.168.203.106@tcp37 [14966.306282] LNet: Removed LNI 192.168.203.106@tcp38 [14966.308560] LNet: Removed LNI 192.168.203.106@tcp39 [14966.311867] LNet: Removed LNI 192.168.203.106@tcp40 [14966.314495] LNet: Removed LNI 192.168.203.106@tcp41 [14966.320908] LNet: Removed LNI 192.168.203.106@tcp42 [14966.324610] LNet: Removed LNI 192.168.203.106@tcp43 [14966.330841] LNet: Removed LNI 192.168.203.106@tcp44 [14966.334565] LNet: Removed LNI 192.168.203.106@tcp45 [14966.338228] LNet: Removed LNI 192.168.203.106@tcp46 [14966.340991] LNet: Removed LNI 192.168.203.106@tcp47 [14966.345781] LNet: Removed LNI 192.168.203.106@tcp48 [14966.350333] LNet: Removed LNI 192.168.203.106@tcp49 [14966.361646] LNet: Removed LNI 192.168.203.106@tcp50 [14967.209792] Key type .llcrypt unregistered [14967.212598] Key type ._llcrypt unregistered [14988.250078] Key type ._llcrypt registered [14988.252191] Key type .llcrypt registered [14988.389113] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [15002.927730] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [15003.803282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15003.962173] alg: No test for adler32 (adler32-zlib) [15005.003760] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [15005.191464] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [15006.855173] Key type lgssc registered [15007.853512] Lustre: Echo OBD driver; http://www.lustre.org/ [15039.485099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15039.506477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15040.720750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15040.746522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15040.828706] Lustre: lustre-MDT0000: new disk, initializing [15040.972218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15041.006650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15045.520373] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15055.074463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15055.194254] Lustre: 505695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15055.245918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15055.250405] Lustre: Skipped 1 previous similar message [15055.361259] Lustre: lustre-MDT0001: new disk, initializing [15055.459248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15055.498805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15055.521613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15059.223351] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15068.281813] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15074.929211] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15083.506284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15083.714811] Lustre: *** cfs_fail_loc=90f, val=0*** [15083.774972] Lustre: lustre-OST0000: new disk, initializing [15083.780702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15083.785542] Lustre: 507714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15083.872043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15090.522438] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15092.306103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15092.315761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15092.453291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15100.097348] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15108.066363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15108.068168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15108.071747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15108.073524] LustreError: Skipped 1 previous similar message [15108.098526] Lustre: Skipped 1 previous similar message [15112.920400] Lustre: server umount lustre-OST0000 complete [15116.768365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15116.774739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15116.789279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15116.799680] Lustre: Skipped 1 previous similar message [15118.308278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15118.309241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15122.509859] Lustre: server umount lustre-MDT0000 complete [15126.156132] LustreError: 505689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859277 with bad export cookie 4639043585751487803 [15126.162565] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15126.175129] LustreError: 505689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15126.508470] Lustre: server umount lustre-MDT0001 complete [15136.596991] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [15138.753573] Key type lgssc unregistered [15138.990397] LNet: 509620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15138.996419] LNetError: 509620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15139.010484] LNet: Removed LNI 192.168.203.106@tcp [15139.753781] Key type .llcrypt unregistered [15139.756109] Key type ._llcrypt unregistered [15154.723977] Key type ._llcrypt registered [15154.727707] Key type .llcrypt registered [15154.799836] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [15155.380268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15155.505190] alg: No test for adler32 (adler32-zlib) [15156.451678] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [15156.595930] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [15158.224884] Key type lgssc registered [15159.223776] Lustre: Echo OBD driver; http://www.lustre.org/ [15165.817104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15165.832595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15167.339720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15170.733791] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15177.676744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15177.700321] LustreError: 510869:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15177.718074] LustreError: 510869:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [15178.045940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15181.874701] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15188.791580] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15193.488263] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15198.712602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15198.996995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15204.388513] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15212.628956] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15222.897348] loop0: detected capacity change from 0 to 204800000 [15223.090375] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15223.099596] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15223.108767] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15227.761152] loop0: detected capacity change from 0 to 204800000 [15227.939669] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15227.950415] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15227.957477] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15232.050582] loop0: detected capacity change from 0 to 204800000 [15236.384696] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15236.521510] Lustre: fs15246-OST0000: new disk, initializing [15236.525434] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15236.536351] Lustre: 514462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15236.593142] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15241.232575] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15249.243847] loop1: detected capacity change from 0 to 204800000 [15254.053120] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15254.166881] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15254.251237] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15254.338882] Lustre: fs15246-MDT0000: new disk, initializing [15254.499133] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15254.537915] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15254.619813] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15255.651818] Lustre: 515582:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000816a2a5 x1874706360740096/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787859422 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15258.779832] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15264.757152] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15264.767304] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15264.786257] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15277.556718] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15277.582392] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15282.135215] Lustre: server umount fs15246-MDT0000 complete [15282.659739] LustreError: 510866:0:(ldlm_lib.c:1212:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15287.777593] LustreError: 512304:0:(ldlm_lib.c:1212:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15290.516987] Lustre: server umount fs15246-OST0000 complete [15303.150527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15303.169231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15303.180269] Lustre: Skipped 1 previous similar message [15308.088172] Lustre: server umount lustre-OST0000 complete [15313.387293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15313.395273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15313.403878] Lustre: Skipped 2 previous similar messages [15318.006856] Lustre: server umount lustre-MDT0000 complete [15318.503345] LustreError: 511934:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15318.525050] LustreError: 511934:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [15321.801512] LustreError: 510851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859472 with bad export cookie 14073524532867434558 [15321.808798] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15321.810973] LustreError: 510851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15322.250045] Lustre: server umount lustre-MDT0001 complete [15332.676875] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [15335.223682] Key type lgssc unregistered [15335.546024] LNet: 518684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15335.557864] LNetError: 518684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15335.575596] LNet: Removed LNI 192.168.203.106@tcp [15336.244389] Key type .llcrypt unregistered [15336.246772] Key type ._llcrypt unregistered [15352.315981] Key type ._llcrypt registered [15352.321641] Key type .llcrypt registered [15352.452912] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [15353.435169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15353.519735] alg: No test for adler32 (adler32-zlib) [15354.526152] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [15354.746958] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [15356.447174] Key type lgssc registered [15357.567582] Lustre: Echo OBD driver; http://www.lustre.org/ [15393.207345] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [15402.595326] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [15434.862805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15434.891323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15436.206902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15436.246519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15436.337547] Lustre: lustre-MDT0000: new disk, initializing [15436.460251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15436.489426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15440.699900] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15450.020511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15450.129949] Lustre: 523149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15450.173373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15450.176061] Lustre: Skipped 1 previous similar message [15450.243827] Lustre: lustre-MDT0001: new disk, initializing [15450.363159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15450.430815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15450.437849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15454.329246] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15465.403168] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15471.268384] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15479.204485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15479.568333] Lustre: lustre-OST0000: new disk, initializing [15479.571840] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15479.587492] Lustre: 525114:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15479.685941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15486.419409] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15486.543573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15486.561637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15486.781330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15496.433976] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15501.135593] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15501.346246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15506.335552] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15506.532781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15512.546432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15512.548963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15512.562328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15512.567226] Lustre: Skipped 1 previous similar message [15514.797794] Lustre: server umount lustre-OST0000 complete [15522.272417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15522.282347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15522.307541] Lustre: Skipped 1 previous similar message [15522.317733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15524.566154] Lustre: server umount lustre-MDT0000 complete [15528.137986] LustreError: 523942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859679 with bad export cookie 13163040185625534589 [15528.149484] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15528.154027] LustreError: 523942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15528.431937] Lustre: server umount lustre-MDT0001 complete [15550.959847] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [15560.401651] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [15589.359198] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15365 sec ========= 15:42:19 (1787859739) [15591.150490] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:42:21 (1787859741) === [15594.343968] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:42:24 (1787859744) === [15631.083693] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [15633.637965] Key type lgssc unregistered [15633.964156] LNet: 531762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15633.973883] LNetError: 531762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15634.980915] LNet: Removed LNI 192.168.203.106@tcp [15635.795549] Key type .llcrypt unregistered [15635.797317] Key type ._llcrypt unregistered