[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 547014818 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003244] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007821] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.011089] LSM: Security Framework initializing [ 0.012077] Yama: becoming mindful. [ 0.014032] SELinux: Initializing. [ 0.015062] *** VALIDATE selinux *** [ 0.024335] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028796] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030055] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033122] *** VALIDATE tmpfs *** [ 0.034463] *** VALIDATE proc *** [ 0.035238] *** VALIDATE cgroup *** [ 0.036009] *** VALIDATE cgroup2 *** [ 0.038120] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041028] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.044764] debug: unmapping init [mem 0xffffffffbe059000-0xffffffffbe060fff] [ 0.046270] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047792] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048021] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.051010] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053012] ... fixed-purpose events: 3 [ 0.054009] ... event mask: 000000070000000f [ 0.055339] rcu: Hierarchical SRCU implementation. [ 0.057614] smp: Bringing up secondary CPUs ... [ 0.058499] x86: Booting SMP configuration: [ 0.059014] .... node #0, CPUs: #1 #2 #3 [ 0.071097] smp: Brought up 1 node, 4 CPUs [ 0.073013] smpboot: Max logical packages: 1 [ 0.074020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.109571] node 0 deferred pages initialised in 33ms [ 0.113005] devtmpfs: initialized [ 0.114130] x86/mm: Memory block size: 128MB [ 0.116159] gcov: version magic: 0x41383552 [ 0.118089] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.119051] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.120181] pinctrl core: initialized pinctrl subsystem [ 0.121126] [ 0.122007] ************************************************************* [ 0.123008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.124012] ** ** [ 0.125010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.126010] ** ** [ 0.127010] ** This means that this kernel is built to expose internal ** [ 0.128009] ** IOMMU data structures, which may compromise security on ** [ 0.129009] ** your system. ** [ 0.130008] ** ** [ 0.131008] ** If you see this message and you are not debugging the ** [ 0.132008] ** kernel, report this immediately to your vendor! ** [ 0.133008] ** ** [ 0.134008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135009] ************************************************************* [ 0.136927] NET: Registered protocol family 16 [ 0.137572] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.138060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.139055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.140423] cpuidle: using governor menu [ 0.143236] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.144422] PCI: Using configuration type 1 for base access [ 0.145143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.153084] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.154015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.156083] cryptd: max_cpu_qlen set to 1000 [ 0.159008] ACPI: Added _OSI(Module Device) [ 0.160011] ACPI: Added _OSI(Processor Device) [ 0.162010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.164012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.169871] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.177987] ACPI: Interpreter enabled [ 0.179238] ACPI: PM: (supports S0 S3 S4 S5) [ 0.181011] ACPI: Using IOAPIC for interrupt routing [ 0.183120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.188530] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.201000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.203045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.205018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.209168] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.214640] acpiphp: Slot [2] registered [ 0.217148] acpiphp: Slot [5] registered [ 0.218217] acpiphp: Slot [6] registered [ 0.220123] acpiphp: Slot [7] registered [ 0.222121] acpiphp: Slot [8] registered [ 0.224102] acpiphp: Slot [9] registered [ 0.226168] acpiphp: Slot [10] registered [ 0.227191] acpiphp: Slot [3] registered [ 0.229083] acpiphp: Slot [4] registered [ 0.231110] acpiphp: Slot [11] registered [ 0.232089] acpiphp: Slot [12] registered [ 0.234156] acpiphp: Slot [13] registered [ 0.236081] acpiphp: Slot [14] registered [ 0.238110] acpiphp: Slot [15] registered [ 0.239082] acpiphp: Slot [16] registered [ 0.241140] acpiphp: Slot [17] registered [ 0.243091] acpiphp: Slot [18] registered [ 0.244080] acpiphp: Slot [19] registered [ 0.245060] acpiphp: Slot [20] registered [ 0.247064] acpiphp: Slot [21] registered [ 0.248083] acpiphp: Slot [22] registered [ 0.250120] acpiphp: Slot [23] registered [ 0.252118] acpiphp: Slot [24] registered [ 0.253099] acpiphp: Slot [25] registered [ 0.255094] acpiphp: Slot [26] registered [ 0.257077] acpiphp: Slot [27] registered [ 0.258084] acpiphp: Slot [28] registered [ 0.259053] acpiphp: Slot [29] registered [ 0.260134] acpiphp: Slot [30] registered [ 0.263184] acpiphp: Slot [31] registered [ 0.264052] PCI host bridge to bus 0000:00 [ 0.266020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.268098] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.270015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.272016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.274014] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.276067] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.278227] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.282650] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.286514] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.299017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.303049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.306015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.308014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.313072] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.317081] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.319869] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.323049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.326987] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.333062] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.347023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.353020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.357601] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.369019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.378017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.406022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.423118] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.438042] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.447032] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.470020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.483286] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.496017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.507016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.539017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.561935] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.577017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.585015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.609017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.622481] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.625000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.625000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.646027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.766960] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.785017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.795024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.846017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.872298] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.875378] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.877616] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.880425] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.883221] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.887177] iommu: Default domain type: Passthrough [ 0.891468] SCSI subsystem initialized [ 0.893165] ACPI: bus type USB registered [ 0.895206] usbcore: registered new interface driver usbfs [ 0.898111] usbcore: registered new interface driver hub [ 0.901204] usbcore: registered new device driver usb [ 0.904208] pps_core: LinuxPPS API ver. 1 registered [ 0.906010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.913126] PTP clock support registered [ 0.915138] EDAC MC: Ver: 3.0.0 [ 0.916120] PCI: Using ACPI for IRQ routing [ 0.918585] NetLabel: Initializing [ 0.920012] NetLabel: domain hash size = 128 [ 0.922009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.924073] NetLabel: unlabeled traffic allowed by default [ 0.927412] vgaarb: loaded [ 0.929594] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.931012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.938571] clocksource: Switched to clocksource kvm-clock [ 1.295939] VFS: Disk quotas dquot_6.6.0 [ 1.297506] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.302898] *** VALIDATE ramfs *** [ 1.304293] *** VALIDATE hugetlbfs *** [ 1.306050] pnp: PnP ACPI init [ 1.308576] pnp: PnP ACPI: found 6 devices [ 1.389163] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.401656] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.407234] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.409444] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.411895] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.414148] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.416912] NET: Registered protocol family 2 [ 1.419561] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.424613] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.428189] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.433975] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.437966] TCP: Hash tables configured (established 65536 bind 65536) [ 1.441213] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.452104] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.454439] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.457120] NET: Registered protocol family 1 [ 1.459279] RPC: Registered named UNIX socket transport module. [ 1.461055] RPC: Registered udp transport module. [ 1.462481] RPC: Registered tcp transport module. [ 1.463952] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.465846] NET: Registered protocol family 44 [ 1.467240] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.468995] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.470830] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.472676] PCI: CLS 0 bytes, default 64 [ 1.474592] Unpacking initramfs... [ 3.431660] debug: unmapping init [mem 0xffff8975fcc54000-0xffff8975fffbffff] [ 3.436627] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.439067] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.442349] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.010091] Initialise system trusted keyrings [ 4.011415] Key type blacklist registered [ 4.013317] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.023447] zbud: loaded [ 4.026473] *** VALIDATE nfs *** [ 4.027685] *** VALIDATE nfs4 *** [ 4.029371] pstore: using deflate compression [ 4.033288] Platform Keyring initialized [ 4.149992] NET: Registered protocol family 38 [ 4.151988] Key type asymmetric registered [ 4.153817] Asymmetric key parser 'x509' registered [ 4.155958] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.159609] io scheduler mq-deadline registered [ 4.161508] io scheduler kyber registered [ 4.162925] io scheduler bfq registered [ 4.169281] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.173794] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.176232] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.179107] ACPI: Power Button [PWRF] [ 4.185099] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.192137] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.290375] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.329657] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.368035] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.401285] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.433862] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.438818] Non-volatile memory driver v1.3 [ 4.440616] Linux agpgart interface v0.103 [ 4.475135] virtio_blk virtio1: [vda] 134864 512-byte logical blocks (69.1 MB/65.9 MiB) [ 4.478790] vda: detected capacity change from 0 to 69050368 [ 4.512369] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.515639] vdb: detected capacity change from 0 to 1073741824 [ 4.530447] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.533370] vdc: detected capacity change from 0 to 2621440000 [ 4.549524] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.552300] vdd: detected capacity change from 0 to 2621440000 [ 4.575404] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.578055] vde: detected capacity change from 0 to 4294967296 [ 4.597251] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.600375] vdf: detected capacity change from 0 to 4294967296 [ 4.618591] libphy: Fixed MDIO Bus: probed [ 4.628667] usbcore: registered new interface driver usbserial_generic [ 4.631316] usbserial: USB Serial support registered for generic [ 4.633980] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.638922] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.640932] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.643559] mousedev: PS/2 mouse device common for all mice [ 4.646587] rtc_cmos 00:05: RTC can wake from S4 [ 4.650255] rtc_cmos 00:05: registered as rtc0 [ 4.652975] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.655562] intel_pstate: CPU model not supported [ 4.658505] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.659641] hid: raw HID events driver (C) Jiri Kosina [ 4.664960] usbcore: registered new interface driver usbhid [ 4.667221] usbhid: USB HID core driver [ 4.668788] drop_monitor: Initializing network drop monitor service [ 4.670970] Initializing XFRM netlink socket [ 4.673933] NET: Registered protocol family 10 [ 4.677282] Segment Routing with IPv6 [ 4.678618] NET: Registered protocol family 17 [ 4.681329] mpls_gso: MPLS GSO support [ 4.685070] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.693906] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.698678] RAS: Correctable Errors collector initialized. [ 4.700893] AVX version of gcm_enc/dec engaged. [ 4.702471] AES CTR mode by8 optimization enabled [ 4.785890] sched_clock: Marking stable (4785810990, 0)->(5984740368, -1198929378) [ 4.790347] registered taskstats version 1 [ 4.792569] Loading compiled-in X.509 certificates [ 4.794823] zswap: loaded using pool lzo/zbud [ 4.825148] Key type big_key registered [ 4.842912] Key type encrypted registered [ 4.845903] ima: No TPM chip found, activating TPM-bypass! [ 4.848463] ima: Allocated hash algorithm: sha1 [ 4.850313] ima: No architecture policies found [ 4.852494] evm: Initialising EVM extended attributes: [ 4.854553] evm: security.selinux [ 4.855760] evm: security.ima [ 4.857122] evm: security.capability [ 4.858586] evm: HMAC attrs: 0x1 [ 4.861468] rtc_cmos 00:05: setting system clock to 2026-06-01 15:13:41 UTC (1780326821) [ 4.869340] debug: unmapping init [mem 0xffffffffbf003000-0xffffffffbf1fffff] [ 4.872959] debug: unmapping init [mem 0xffffffffbdd82000-0xffffffffbe058fff] [ 4.881190] Write protecting the kernel read-only data: 28672k [ 4.885215] debug: unmapping init [mem 0xffffffffbc403000-0xffffffffbc5fffff] [ 4.888371] debug: unmapping init [mem 0xffffffffbcd14000-0xffffffffbcdfffff] [ 4.928457] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.940151] systemd[1]: Detected virtualization kvm. [ 4.942553] systemd[1]: Detected architecture x86-64. [ 4.944786] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.972700] systemd[1]: No hostname configured. [ 4.974783] systemd[1]: Set hostname to . [ 4.976615] random: systemd: uninitialized urandom read (16 bytes read) [ 4.978868] systemd[1]: Initializing machine ID from random generator. [ 5.070525] random: ln: uninitialized urandom read (6 bytes read) [ 5.907698] random: systemd: uninitialized urandom read (16 bytes read) [ 5.912583] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.924936] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.000988] systemd[1]: Starting Journal Service... Starting Journal Service... [ 6.020583] urandom_read: 2 callbacks suppressed [ 6.020594] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Reached target Local File Systems. [ 6.039418] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Started Memstrack Anylazing Service. [ 6.081509] random: systemd: uninitialized urandom read (16 bytes read) Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.405951] device-mapper: uevent: version 1.0.3 [ 7.408572] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 8.927970] random: fast init done [ 9.068642] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started udev Coldplug all Devices. [ 9.109334] scsi host0: ata_piix [ 9.121673] scsi host1: ata_piix [ 9.123591] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.126578] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 15.740944] random: crng init done [ 15.753612] random: 2 urandom warning(s) missed due to ratelimiting [ 16.065772] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.360466] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.093760] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.502321] SELinux: Disabled at runtime. [ 20.607910] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.621364] systemd[1]: Detected virtualization kvm. [ 20.624323] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.498758] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.502727] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.514350] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.518918] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.522759] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.529451] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.536838] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ 22.612991] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 24.746995] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.466292] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.602374] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.033455] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.455615] EDAC sbridge: Ver: 1.1.2 [ 30.876495] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (9s / no limit)[ 31.587344] NFS: Registering the id_resolver key type [ 31.591159] Key type id_resolver registered [ 31.593636] Key type id_legacy registered [** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg220-server login: [ 87.993949] libcfs: loading out-of-tree module taints kernel. [ 88.025486] Key type ._llcrypt registered [ 88.026662] Key type .llcrypt registered [ 88.121168] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 103.238669] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 104.435922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 104.451835] alg: No test for adler32 (adler32-zlib) [ 105.965767] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 106.809417] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 108.543366] Key type lgssc registered [ 109.798475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 124.040774] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 155.810185] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 166.852722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 166.908471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 168.181186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 168.216952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 168.310570] Lustre: lustre-MDT0000: new disk, initializing [ 168.380746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 168.394748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 169.647017] hrtimer: interrupt took 6324695 ns [ 171.750173] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.752987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.872279] Lustre: 6503:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.888875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 182.896362] Lustre: Skipped 1 previous similar message [ 183.010201] Lustre: lustre-MDT0001: new disk, initializing [ 183.066230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 183.083964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 183.094770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.497197] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.580545] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 199.087918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.384228] Lustre: lustre-OST0000: new disk, initializing [ 199.387676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.395703] Lustre: 8408:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.453543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.456713] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.380475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.391506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 207.438186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 216.676816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.794568] Lustre: lustre-OST0001: new disk, initializing [ 216.797867] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.802551] Lustre: 9464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.843474] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.962411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 216.971703] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 217.040184] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 222.145602] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.733059] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.001150] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.598345] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing check_logdir /tmp/testlogs/ [ 251.466767] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing yml_node [ 254.482430] Lustre: DEBUG MARKER: Client: 2.17.53.33 [ 256.731823] Lustre: DEBUG MARKER: MDS: 2.17.53.33 [ 258.974697] Lustre: DEBUG MARKER: OSS: 2.17.53.33 [ 260.412481] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jun 1 11:17:55 EDT 2026 [ 274.842408] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 275.970954] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 290.784801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 290.800100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.817764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.860453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.866420] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.885236] Lustre: Skipped 1 previous similar message [ 296.732142] Lustre: server umount lustre-MDT0000 complete [ 299.747606] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327116 with bad export cookie 3277986596943253120 [ 299.752447] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 299.753549] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 304.096324] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.099789] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.102527] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 304.102537] Lustre: Skipped 3 previous similar messages [ 304.103042] Lustre: Skipped 2 previous similar messages [ 304.110388] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 309.215669] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.216974] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 309.224415] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 309.227427] Lustre: Skipped 2 previous similar messages [ 310.241036] LustreError: 7955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.262019] LustreError: 7955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 314.335194] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 314.489628] Lustre: server umount lustre-MDT0001 complete [ 317.739937] Lustre: server umount lustre-OST0000 complete [ 320.511215] Lustre: server umount lustre-OST0001 complete [ 325.267948] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 332.341735] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 363.624202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 363.842609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 363.871842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 363.925930] Lustre: lustre-MDT0000: new disk, initializing [ 364.027615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.042775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 367.517606] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 376.578888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.674936] Lustre: 14978:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 376.743940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 376.750698] Lustre: Skipped 1 previous similar message [ 376.867835] Lustre: lustre-MDT0001: new disk, initializing [ 376.953745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 376.996370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 377.012069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 380.336086] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 387.579440] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 391.396884] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 396.995220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 397.269662] Lustre: lustre-OST0000: new disk, initializing [ 397.273798] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 397.279136] Lustre: 16721:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 397.362765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 398.779459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 398.785333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 398.927960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 402.562701] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 409.469451] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 413.384420] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 413.529596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 417.382288] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 417.577755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 424.420502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 424.421649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 424.431158] LustreError: Skipped 1 previous similar message [ 424.433919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 424.455285] Lustre: Skipped 1 previous similar message [ 424.467211] Lustre: Skipped 3 previous similar messages [ 425.624145] Lustre: server umount lustre-OST0000 complete [ 429.543520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 429.552236] Lustre: Skipped 1 previous similar message [ 434.658523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 434.673021] Lustre: Skipped 3 previous similar messages [ 443.361083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 443.550280] Lustre: server umount lustre-MDT0000 complete [ 444.896749] LustreError: 14989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.928472] LustreError: 14989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 446.733905] LustreError: 15758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327263 with bad export cookie 3277986596943254695 [ 446.734364] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 446.742565] LustreError: 15758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 447.109529] Lustre: server umount lustre-MDT0001 complete [ 453.267556] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 11:21:08 (1780327268) [ 469.643549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.014238] LustreError: 19452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 470.088677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.826679] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 480.241995] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.262413] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 480.665710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.158761] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 488.581643] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 491.618778] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 493.400663] Lustre: Failing over lustre-MDT0000 [ 495.585636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.640852] Lustre: server umount lustre-MDT0000 complete [ 500.704646] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.721241] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 511.967164] Lustre: 3658:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780327312/real 1780327312] req@ffff89765ec30e00 x1866808083768704/t0(0) o400->MGC192.168.202.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780327328 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 512.008483] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 513.825035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.214381] LustreError: 3655:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff897542253100 x1866808083771520/t0(0) o250->MGC192.168.202.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 525.542050] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 525.546200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 525.559903] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 525.727790] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 531.933771] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 536.034175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 536.040065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.052721] Lustre: Skipped 1 previous similar message [ 536.056595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 536.060263] Lustre: Skipped 2 previous similar messages [ 541.606107] Lustre: server umount lustre-MDT0000 complete [ 543.204251] LustreError: 20177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.225111] LustreError: 20177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 544.807140] LustreError: 19434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327361 with bad export cookie 3277986596943256151 [ 544.814240] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.998219] Lustre: server umount lustre-MDT0001 complete [ 552.381028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.746109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 552.752182] Lustre: Skipped 2 previous similar messages [ 556.235472] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 562.986212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.461487] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 571.363455] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 574.583881] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 576.293435] Lustre: Failing over lustre-MDT0000 [ 576.434560] Lustre: server umount lustre-MDT0000 complete [ 577.504138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.515894] Lustre: Skipped 2 previous similar messages [ 593.693557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.804933] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.122541] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 599.525414] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 599.527085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 599.534176] Lustre: Skipped 1 previous similar message [ 599.556766] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 603.523194] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 607.651386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 607.654386] Lustre: Skipped 2 previous similar messages [ 612.948787] Lustre: server umount lustre-MDT0000 complete [ 615.905418] LustreError: 22433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327432 with bad export cookie 3277986596943257089 [ 615.911152] LustreError: 22433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 615.916034] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 616.082231] Lustre: server umount lustre-MDT0001 complete [ 622.312951] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 11:23:57 (1780327437) [ 634.998470] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 642.161119] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 671.222396] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 678.727345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.877211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 678.900817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 678.973808] Lustre: lustre-MDT0000: new disk, initializing [ 679.031646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 681.586535] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 688.337875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 688.417697] Lustre: 28430:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 688.566125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 688.571659] Lustre: Skipped 3 previous similar messages [ 688.593156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 691.131510] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 696.077631] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 698.234698] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 703.034973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 703.269960] Lustre: 30171:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 704.965751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 705.018086] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 705.082909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 705.090152] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 707.755572] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 712.411489] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 717.443689] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 730.594202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 730.594454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 730.602849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 730.609682] Lustre: Skipped 3 previous similar messages [ 730.620017] Lustre: Skipped 2 previous similar messages [ 736.297336] Lustre: server umount lustre-OST0000 complete [ 739.809457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 739.815721] LustreError: Skipped 1 previous similar message [ 745.955430] LustreError: 30295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 745.965018] LustreError: 30295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 747.399952] Lustre: server umount lustre-MDT0000 complete [ 750.004941] LustreError: 28422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327566 with bad export cookie 3277986596943257495 [ 750.009239] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 750.016096] LustreError: 28422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 750.267801] Lustre: server umount lustre-MDT0001 complete [ 756.810274] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 758.512899] Key type lgssc unregistered [ 758.738644] LNet: 32017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.743797] LNetError: 32017:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.757158] LNet: Removed LNI 192.168.202.120@tcp [ 759.313195] Key type .llcrypt unregistered [ 759.314875] Key type ._llcrypt unregistered [ 775.797423] Key type ._llcrypt registered [ 775.798937] Key type .llcrypt registered [ 775.874883] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 776.646729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 776.669960] alg: No test for adler32 (adler32-zlib) [ 777.751597] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 777.939855] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 779.591640] Key type lgssc registered [ 780.326351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 785.352301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 785.378361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 786.681398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 789.314737] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 795.112851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.317965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 798.115539] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 801.270459] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 802.982434] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 807.243724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 807.567833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.339179] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 814.693390] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 819.894879] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 827.736470] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 11:27:23 (1780327643) [ 829.033483] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 830.219774] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 834.100684] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 11:27:29 (1780327649) [ 839.312916] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 11:27:35 (1780327655) [ 840.004884] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 845.339382] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 845.345424] Lustre: Skipped 2 previous similar messages [ 846.384269] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 11:27:42 (1780327662) [ 850.941322] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 850.946228] Lustre: Skipped 4 previous similar messages [ 859.075279] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 859.077995] Lustre: Skipped 8 previous similar messages [ 860.111260] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 11:27:55 (1780327675) [ 861.689308] Lustre: *** cfs_fail_loc=131b, val=2*** [ 863.864634] Lustre: *** cfs_fail_loc=131b, val=2*** [ 868.970870] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 11:28:04 (1780327684) [ 876.934272] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 876.938955] Lustre: Skipped 4 previous similar messages [ 877.993442] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 11:28:13 (1780327693) [ 886.755663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 886.757181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 886.767394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 886.773587] Lustre: Skipped 1 previous similar message [ 891.497587] Lustre: server umount lustre-OST0000 complete [ 896.994083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 896.999665] Lustre: Skipped 1 previous similar message [ 897.003708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 897.007949] Lustre: Skipped 1 previous similar message [ 899.715955] Lustre: server umount lustre-MDT0000 complete [ 901.708761] LustreError: 37573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327718 with bad export cookie 2691845578722668031 [ 901.711983] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 901.715548] LustreError: 37573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 901.894841] Lustre: server umount lustre-MDT0001 complete [ 907.504813] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 909.055379] Key type lgssc unregistered [ 909.255780] LNet: 40872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.259934] LNetError: 40872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.271797] LNet: Removed LNI 192.168.202.120@tcp [ 909.669125] Key type .llcrypt unregistered [ 909.671068] Key type ._llcrypt unregistered [ 919.168182] Key type ._llcrypt registered [ 919.170825] Key type .llcrypt registered [ 919.220138] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 928.338660] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 929.003324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 929.019976] alg: No test for adler32 (adler32-zlib) [ 929.943507] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 930.045889] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 931.647191] Key type lgssc registered [ 932.384851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 953.141109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 953.164805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.335345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 954.356460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 954.401941] Lustre: lustre-MDT0000: new disk, initializing [ 954.458148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.467861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 956.652471] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 962.916209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.008863] Lustre: 43969:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 963.058155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 963.061508] Lustre: Skipped 1 previous similar message [ 963.110443] Lustre: lustre-MDT0001: new disk, initializing [ 963.174461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 963.217560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 963.232391] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 965.577979] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 970.320473] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 972.493120] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 977.146160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 977.354380] Lustre: lustre-OST0000: new disk, initializing [ 977.357484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 977.361911] Lustre: 45711:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 977.428238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 981.698687] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 983.065334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 983.074981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 983.129369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 986.816950] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 989.693284] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 989.833731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 992.270471] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 992.383459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 998.369711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 998.370138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.373410] LustreError: Skipped 1 previous similar message [ 998.384940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 998.390296] Lustre: Skipped 1 previous similar message [ 999.775981] Lustre: server umount lustre-OST0000 complete [ 1003.488646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1003.497962] Lustre: Skipped 1 previous similar message [ 1003.501156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1003.503769] Lustre: Skipped 1 previous similar message [ 1008.612080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1008.620864] Lustre: Skipped 1 previous similar message [ 1013.728508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1013.732094] Lustre: Skipped 1 previous similar message [ 1016.288284] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1016.898855] Lustre: server umount lustre-MDT0000 complete [ 1022.315159] LustreError: 43961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327838 with bad export cookie 1096012425368262148 [ 1022.323809] LustreError: 43961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1022.324680] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1023.035590] Lustre: server umount lustre-MDT0001 complete [ 1040.175187] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 11:30:54 (1780327854) [ 1051.213054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.419212] LustreError: 48025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1052.450812] LustreError: 48025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1052.570828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.769072] LustreError: 48026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1059.698442] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1062.880351] LustreError: 48025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1068.005653] LustreError: 48026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.080528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.874098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1079.375955] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1087.232553] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1092.894272] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1102.374409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1103.003727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.139426] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1115.932504] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1130.633463] LustreError: 50239:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1131.676486] Lustre: Modifying parameter general.timeout=1 in log params [ 1133.400585] Lustre: Modifying parameter general.timeout=3 in log params [ 1133.404318] Lustre: Skipped 1 previous similar message [ 1135.983733] Lustre: Modifying parameter general.timeout=6 in log params [ 1135.986339] Lustre: Skipped 2 previous similar messages [ 1140.221196] Lustre: Modifying parameter general.timeout=10 in log params [ 1140.228083] Lustre: Skipped 3 previous similar messages [ 1148.692947] Lustre: Modifying parameter general.timeout=19 in log params [ 1148.699655] Lustre: Skipped 8 previous similar messages [ 1165.059312] Lustre: Modifying parameter general.timeout=37 in log params [ 1165.062557] Lustre: Skipped 17 previous similar messages [ 1197.757107] Lustre: Modifying parameter general.timeout=72 in log params [ 1197.761122] Lustre: Skipped 34 previous similar messages [ 1261.883807] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 11:34:36 (1780328076) [ 1262.911552] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1262.917201] Lustre: Skipped 58 previous similar messages [ 1264.723489] Lustre: 57181:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1266.686690] Lustre: 57279:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1270.522598] Lustre: 57495:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1270.531562] Lustre: 57495:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1301.923711] Lustre: 59053:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1301.930255] Lustre: 59053:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1301.965153] LustreError: 59053:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1306.068610] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 11:35:21 (1780328121) [ 1318.880933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1318.888100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.903905] Lustre: Skipped 1 previous similar message [ 1318.919744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.513944] Lustre: server umount lustre-MDT0000 complete [ 1325.026215] LustreError: 48026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1325.046511] LustreError: 48026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1326.127437] LustreError: 58379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328142 with bad export cookie 1096012425368263058 [ 1326.130452] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1326.144628] LustreError: 58379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1326.567531] Lustre: server umount lustre-MDT0001 complete [ 1341.177900] Lustre: server umount lustre-OST0000 complete [ 1370.470434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1370.890470] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1371.045927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.771289] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1387.989805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.042409] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1388.088299] Lustre: 61584:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1395.212365] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1404.411123] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1410.965301] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1419.832917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1420.104414] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1420.526672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.531944] Lustre: Skipped 1 previous similar message [ 1428.292444] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1436.448422] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1439.392852] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1439.407795] Lustre: Skipped 22 previous similar messages [ 1439.468631] Lustre: 63815:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1452.515247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1452.529602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.552357] Lustre: Skipped 3 previous similar messages [ 1452.592187] Lustre: Skipped 4 previous similar messages [ 1456.834840] Lustre: server umount lustre-OST0000 complete [ 1465.313125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1465.326184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.472270] Lustre: server umount lustre-MDT0000 complete [ 1472.997446] LustreError: 61597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.026728] LustreError: 61597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1474.011968] LustreError: 61576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328290 with bad export cookie 1096012425368266089 [ 1474.025522] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1474.032490] LustreError: 61576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1474.724616] Lustre: server umount lustre-MDT0001 complete [ 1488.579419] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1492.369997] Key type lgssc unregistered [ 1492.777545] LNet: 65010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.787654] LNetError: 65010:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1493.866622] LNet: Removed LNI 192.168.202.120@tcp [ 1495.217191] Key type .llcrypt unregistered [ 1495.218996] Key type ._llcrypt unregistered [ 1523.319699] Key type ._llcrypt registered [ 1523.322518] Key type .llcrypt registered [ 1523.439698] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1524.307895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1524.342108] alg: No test for adler32 (adler32-zlib) [ 1525.370747] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 1525.563812] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1527.223278] Key type lgssc registered [ 1528.741196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1536.661406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1536.681298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.177402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1542.917365] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1547.745087] LustreError: 66412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1547.755985] LustreError: 66412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1552.865252] LustreError: 66411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1552.943810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1553.494943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1558.512784] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1564.659659] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1568.665863] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1576.125822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1576.881852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.139664] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1590.327681] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1598.667437] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1608.682150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.695194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1608.709414] Lustre: Skipped 2 previous similar messages [ 1608.725367] Lustre: Skipped 2 previous similar messages [ 1613.803878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1614.071480] Lustre: server umount lustre-MDT0000 complete [ 1618.013979] LustreError: 66393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328434 with bad export cookie 2116657452452804872 [ 1618.022889] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1618.035237] LustreError: 66393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1618.319581] Lustre: server umount lustre-MDT0001 complete [ 1633.593987] Lustre: server umount lustre-OST0000 complete [ 1662.631559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.968650] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1663.002697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1663.363605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1668.396706] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1679.111977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.181977] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1679.220535] Lustre: 70949:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1679.850840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1685.015302] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1692.082770] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1696.970742] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1705.459326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1705.625756] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1706.182784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1712.986760] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1719.233574] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1721.876092] Lustre: 73186:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1724.111271] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1733.603850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1733.611181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.630101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1733.644267] Lustre: Skipped 2 previous similar messages [ 1734.114250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1736.223265] Lustre: server umount lustre-OST0000 complete [ 1741.283656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1741.293063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.311849] Lustre: Skipped 1 previous similar message [ 1741.320847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1741.329290] Lustre: Skipped 2 previous similar messages [ 1744.352768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1747.453451] Lustre: server umount lustre-MDT0000 complete [ 1749.476340] LustreError: 70960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1749.529280] LustreError: 70960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1751.911659] LustreError: 73188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328568 with bad export cookie 2116657452452805719 [ 1751.913060] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1751.923958] LustreError: 73188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1752.593722] Lustre: server umount lustre-MDT0001 complete [ 1766.163802] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1769.664835] Key type lgssc unregistered [ 1769.987701] LNet: 74381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.999157] LNetError: 74381:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1770.036349] LNet: Removed LNI 192.168.202.120@tcp [ 1771.245088] Key type .llcrypt unregistered [ 1771.247269] Key type ._llcrypt unregistered [ 1798.457213] Key type ._llcrypt registered [ 1798.459545] Key type .llcrypt registered [ 1798.597265] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1799.491669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1799.640366] alg: No test for adler32 (adler32-zlib) [ 1800.791146] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 1801.014920] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1802.671546] Key type lgssc registered [ 1803.807798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1811.631511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1811.657750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.349326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.864714] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1822.690992] LustreError: 75777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1822.709351] LustreError: 75777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1827.159261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.702902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1831.976612] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1837.341423] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1840.976874] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1847.032110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1847.621102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.422342] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1858.992686] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1864.461944] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1865.328028] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1867.240835] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1867.249349] Lustre: Skipped 1 previous similar message [ 1869.603930] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1869.606900] Lustre: Skipped 2 previous similar messages [ 1873.735287] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1873.740205] Lustre: Skipped 4 previous similar messages [ 1882.441871] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1882.446178] Lustre: Skipped 10 previous similar messages [ 1898.461752] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1898.467680] Lustre: Skipped 19 previous similar messages [ 1931.757493] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1931.762486] Lustre: Skipped 34 previous similar messages [ 1967.460285] Lustre: 83541:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1970.878545] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 11:46:25 (1780328785) [ 1972.385353] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1974.342343] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 11:46:29 (1780328789) [ 1980.903660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1980.908447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.920474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1980.925110] Lustre: Skipped 1 previous similar message [ 1980.943798] Lustre: Skipped 1 previous similar message [ 1984.495686] Lustre: server umount lustre-OST0000 complete [ 1991.146882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1991.151209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1991.151867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1991.151872] LustreError: Skipped 1 previous similar message [ 1991.163691] Lustre: Skipped 1 previous similar message [ 1991.178598] Lustre: Skipped 1 previous similar message [ 1994.798900] Lustre: server umount lustre-MDT0000 complete [ 1996.256805] LustreError: 75772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.275851] LustreError: 75772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1998.415787] LustreError: 76502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328815 with bad export cookie 7616767316858779741 [ 1998.420103] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1998.434888] LustreError: 76502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1998.784446] Lustre: server umount lustre-MDT0001 complete [ 2008.527696] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 2010.873413] Key type lgssc unregistered [ 2011.194531] LNet: 84773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.202051] LNetError: 84773:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.216305] LNet: Removed LNI 192.168.202.120@tcp [ 2012.008716] Key type .llcrypt unregistered [ 2012.012403] Key type ._llcrypt unregistered [ 2015.666555] Key type ._llcrypt registered [ 2015.674535] Key type .llcrypt registered [ 2015.777726] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2019.777811] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules [ 2020.738806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2020.759523] alg: No test for adler32 (adler32-zlib) [ 2021.942406] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 2022.171466] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 2023.873908] Key type lgssc registered [ 2024.770554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2025.877262] LustreError: 85942:0:(lu_object.c:1479:lu_context_key_register()) cfs_race id 60d sleeping [ 2042.256839] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2048.091560] LustreError: 85942:0:(lu_object.c:1479:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2050.784529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2050.814671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2052.502587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.590466] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2061.812397] LustreError: 86948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.824656] LustreError: 86948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2063.709495] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 11:47:59 (1780328879) [ 2066.915809] LustreError: 86949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2068.064877] LustreError: 86980:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2068.071467] LustreError: 86980:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2068.075732] LustreError: 86980:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 2068.320209] Lustre: server umount lustre-MDT0000 complete [ 2079.198399] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 2081.465955] Key type lgssc unregistered [ 2081.654795] LNet: 88440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2081.659659] LNetError: 88440:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2081.676122] LNet: Removed LNI 192.168.202.120@tcp [ 2082.304460] Key type .llcrypt unregistered [ 2082.308452] Key type ._llcrypt unregistered [ 2098.341602] Key type ._llcrypt registered [ 2098.344649] Key type .llcrypt registered [ 2098.460983] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2099.353522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2099.536160] alg: No test for adler32 (adler32-zlib) [ 2100.592982] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 2100.869527] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 2102.479190] Key type lgssc registered [ 2103.465608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2110.337416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2110.350378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.755348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.583762] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2122.218932] LustreError: 89644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.227044] LustreError: 89644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2124.843203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.627159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2130.313423] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2135.726488] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2138.856798] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2144.903487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2145.504503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.499586] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2156.361722] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2196.552726] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 11:50:12 (1780329012) [ 2197.914613] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2199.603809] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 11:50:15 (1780329015) [ 2203.108983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.113685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2203.115505] Lustre: Skipped 2 previous similar messages [ 2203.122297] Lustre: Skipped 1 previous similar message [ 2208.226212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2208.233782] Lustre: Skipped 2 previous similar messages [ 2208.761624] Lustre: server umount lustre-MDT0000 complete [ 2211.334268] LustreError: 89624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329027 with bad export cookie 3044212111239942116 [ 2211.340152] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2211.345144] LustreError: 89624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2211.603389] Lustre: server umount lustre-MDT0001 complete [ 2225.329855] Lustre: server umount lustre-OST0000 complete [ 2232.919705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.186813] LustreError: 93576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.312862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2236.199897] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2238.436543] LustreError: 93577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2241.298201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.688610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2244.006051] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2247.059949] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2249.099771] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2255.592361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2255.680323] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2255.683293] LustreError: 93565:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2255.697881] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2255.701279] LustreError: 95434:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -98 [ 2255.703571] LustreError: 95434:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2255.707332] LustreError: 95434:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2255.742616] Lustre: server umount lustre-OST0000 complete [ 2255.745123] LustreError: 95434:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2260.262057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2260.383894] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2260.389660] LustreError: 93565:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2260.408580] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2260.411562] LustreError: 95900:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -98 [ 2260.414964] LustreError: 95900:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2260.419829] LustreError: 95900:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2260.453347] Lustre: server umount lustre-OST0000 complete [ 2260.455577] LustreError: 95900:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2265.843390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2265.970704] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2265.973834] Lustre: Found index 0 for lustre-OST0000, updating log [ 2265.977222] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2266.042754] Lustre: lustre-OST0000: new disk, initializing [ 2266.047884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2266.053862] Lustre: 96437:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2266.193212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.606657] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2271.878988] Lustre: Failing over lustre-OST0000 [ 2271.974241] Lustre: server umount lustre-OST0000 complete [ 2274.224929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.228086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2274.237926] Lustre: Skipped 1 previous similar message [ 2274.248861] Lustre: Skipped 1 previous similar message [ 2279.393367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2279.398821] Lustre: Skipped 1 previous similar message [ 2280.485940] Lustre: server umount lustre-MDT0000 complete [ 2282.953815] LustreError: 93558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329099 with bad export cookie 3044212111239944860 [ 2282.956376] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2282.961081] LustreError: 93558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2293.366876] Lustre: server umount lustre-MDT0001 complete [ 2299.461539] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 11:51:55 (1780329115) [ 2308.832493] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2315.065186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.338509] LustreError: 98746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.416952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.613805] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2320.868424] LustreError: 98747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.162761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.354926] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2328.118694] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2332.421334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2332.796318] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2333.798823] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2333.826183] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2333.828991] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2336.182276] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2341.599979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2341.664185] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2341.716161] Lustre: lustre-OST0001: new disk, initializing [ 2341.719587] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2341.725095] Lustre: 101014:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2343.858796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2343.866643] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2343.905505] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2345.201969] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2350.362783] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2355.571507] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2359.276682] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.281065] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2359.292335] Lustre: Skipped 2 previous similar messages [ 2359.296672] Lustre: Skipped 1 previous similar message [ 2364.649946] Lustre: server umount lustre-MDT0001 complete [ 2367.495468] LustreError: 98742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.103866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.206537] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2371.214963] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2371.257201] Lustre: 98733:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2371.717203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2371.721842] Lustre: Skipped 3 previous similar messages [ 2373.170953] LustreError: 102612:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2373.178340] LustreError: 102612:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2373.184385] LustreError: 98739:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2373.337486] LustreError: 102620:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.120@tcp: cfg command failed: rc = -17 [ 2373.345750] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2373.345750] [ 2374.759884] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2377.199088] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2380.299735] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 11:53:16 (1780329196) [ 2391.046842] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2402.494886] Lustre: 104571:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2416.401340] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2418.657505] Lustre: 105703:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2484.192628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2484.199593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.206756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2484.211347] Lustre: Skipped 5 previous similar messages [ 2487.221909] Lustre: server umount lustre-MDT0000 complete [ 2489.541128] LustreError: 101012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329306 with bad export cookie 3044212111239945861 [ 2489.542952] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2489.547273] LustreError: 101012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2489.835773] Lustre: server umount lustre-MDT0001 complete [ 2502.754704] Lustre: server umount lustre-OST0000 complete [ 2515.439959] Lustre: server umount lustre-OST0001 complete [ 2519.036540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2524.560617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2529.189956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2534.193991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2543.927029] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2549.146694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.158134] Lustre: lustre-MDT0000: reset Object Index mappings [ 2549.392214] LustreError: 109636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.399030] LustreError: 109636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2549.441295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2551.234555] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2554.711265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.720258] Lustre: lustre-MDT0001: reset Object Index mappings [ 2556.610215] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2557.889646] Lustre: 110748:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2560.652300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2563.617845] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2565.158530] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 2567.376803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2570.356859] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2572.785241] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2574.836987] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2574.840210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2580.109659] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2582.046206] Lustre: 112591:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2610.920351] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 11:57:06 (1780329426) [ 2641.384346] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.387197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2641.392389] Lustre: Skipped 6 previous similar messages [ 2641.397609] Lustre: Skipped 6 previous similar messages [ 2644.983589] Lustre: server umount lustre-MDT0000 complete [ 2646.499049] LustreError: 109637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2646.509393] LustreError: 109637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2647.364841] LustreError: 110367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329464 with bad export cookie 3044212111239992712 [ 2647.366922] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2647.371804] LustreError: 110367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2651.619775] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.627398] Lustre: Skipped 1 previous similar message [ 2653.778469] Lustre: server umount lustre-MDT0001 complete [ 2662.314381] Lustre: server umount lustre-OST0000 complete [ 2670.544774] Lustre: server umount lustre-OST0001 complete [ 2674.248500] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 2680.237528] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2701.121537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2701.296264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2701.316559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2701.368194] Lustre: lustre-MDT0000: new disk, initializing [ 2701.419332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2701.423031] Lustre: Skipped 3 previous similar messages [ 2701.433540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2703.812983] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2708.445136] Lustre: server umount lustre-MDT0000 complete [ 2714.184546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.354616] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2714.365426] Lustre: 116798:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2714.376177] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2716.911944] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2720.434193] Lustre: server umount lustre-MDT0000 complete [ 2725.303522] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 11:59:01 (1780329541) [ 2726.051542] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2727.058621] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 11:59:03 (1780329543) [ 2735.979345] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2741.182147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.501689] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2748.538090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.596652] Lustre: 118815:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2748.624442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2748.627352] Lustre: Skipped 1 previous similar message [ 2748.677682] Lustre: lustre-MDT0001: new disk, initializing [ 2748.750619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2748.758330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2750.929945] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2753.817084] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2756.906194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.089742] Lustre: lustre-OST0000: new disk, initializing [ 2757.092640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2757.096439] Lustre: 120361:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2757.471711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2757.475774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2757.547565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2760.202444] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2765.736109] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2765.848578] Lustre: lustre-OST0001: new disk, initializing [ 2765.851596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2765.856096] Lustre: 121213:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2767.711293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2767.716328] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2767.764884] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2768.680138] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2774.247792] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2775.971545] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3143.836946] Lustre: 127624:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3143.844511] Lustre: 127624:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3143.850383] Lustre: 127624:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3143.855845] Lustre: 127624:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3143.860894] Lustre: 127624:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3143.865970] Lustre: 127624:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3453.638992] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:11:08 (1780330268) [ 3470.278828] Lustre: server umount lustre-MDT0000 complete [ 3470.821028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3470.828365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3470.836921] LustreError: 118828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.857438] LustreError: 118828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3473.469455] LustreError: 118808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330290 with bad export cookie 3044212111240053066 [ 3473.484114] LustreError: 118808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3473.490932] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3477.476330] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3477.479103] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3477.485634] Lustre: Skipped 4 previous similar messages [ 3477.501363] Lustre: Skipped 5 previous similar messages [ 3479.969172] Lustre: server umount lustre-MDT0001 complete [ 3489.730637] Lustre: server umount lustre-OST0000 complete [ 3507.167169] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3507.328745] Lustre: server umount lustre-OST0001 complete [ 3513.814370] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 3522.657175] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3556.638814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.918898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3556.945859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3557.024486] Lustre: lustre-MDT0000: new disk, initializing [ 3557.107866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3557.114171] Lustre: Skipped 5 previous similar messages [ 3557.125500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3560.543077] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3570.373649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.518239] Lustre: 130649:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3570.593926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3570.598682] Lustre: Skipped 1 previous similar message [ 3570.697781] Lustre: lustre-MDT0001: new disk, initializing [ 3570.819744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3570.833523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3575.354834] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3582.231271] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3585.989738] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3591.797130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3592.087583] Lustre: lustre-OST0000: new disk, initializing [ 3592.091744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3592.097754] Lustre: 132391:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3594.099374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3594.109149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3594.201198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3596.969840] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3602.950713] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3610.368412] Lustre: lustre-MDD0000: changelog on [ 3612.162448] Lustre: lustre-MDD0001: changelog on [ 5760.559808] Lustre: lustre-MDD0001: changelog off [ 5763.129651] Lustre: 130654:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5764.218635] Lustre: lustre-MDD0000: changelog off [ 5775.352384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5775.352462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5775.359292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5775.359304] Lustre: Skipped 2 previous similar messages [ 5775.370352] LustreError: Skipped 1 previous similar message [ 5775.421976] Lustre: Skipped 1 previous similar message [ 5780.450546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5780.471147] Lustre: Skipped 2 previous similar messages [ 5785.569751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5785.587853] Lustre: Skipped 1 previous similar message [ 5787.103163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5787.336511] Lustre: server umount lustre-OST0000 complete [ 5792.739696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5792.758678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5792.776958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5795.833335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5799.008224] Lustre: server umount lustre-MDT0000 complete [ 5800.929047] LustreError: 135809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.962668] LustreError: 135809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5803.346171] LustreError: 131426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332619 with bad export cookie 3044212111240831235 [ 5803.351479] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5803.359606] LustreError: 131426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5803.728760] Lustre: server umount lustre-MDT0001 complete [ 5818.073501] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 5820.859125] Key type lgssc unregistered [ 5821.252451] LNet: 141427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5821.261458] LNetError: 141427:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5821.279297] LNet: Removed LNI 192.168.202.120@tcp [ 5822.358214] Key type .llcrypt unregistered [ 5822.359893] Key type ._llcrypt unregistered [ 5845.654587] Key type ._llcrypt registered [ 5845.659249] Key type .llcrypt registered [ 5845.756066] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 5864.163576] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 5865.709859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5865.758235] alg: No test for adler32 (adler32-zlib) [ 5867.263902] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 5867.653095] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 5869.457964] Key type lgssc registered [ 5871.110860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5921.393551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5921.469122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5922.975731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5923.015752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5923.194385] Lustre: lustre-MDT0000: new disk, initializing [ 5923.402597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5923.495264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5930.151846] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5942.380131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5942.533435] Lustre: 144610:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5942.597742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5942.603800] Lustre: Skipped 1 previous similar message [ 5942.698923] Lustre: lustre-MDT0001: new disk, initializing [ 5942.814891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5942.883856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5942.894420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5947.172096] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5957.089285] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5961.009163] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5969.413051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5969.902608] Lustre: lustre-OST0000: new disk, initializing [ 5969.908738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5969.915447] Lustre: 146352:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5970.061490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5978.437632] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5979.787965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5979.804157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5980.081153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5988.088948] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5994.673659] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_module obdecho/obdecho [ 5996.828748] LustreError: 147224:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5996.850187] LustreError: 147224:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5999.464510] LustreError: 147321:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6014.434111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6014.448548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6014.471739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6015.978619] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6015.979367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6015.991110] Lustre: Skipped 1 previous similar message [ 6016.001365] Lustre: Skipped 2 previous similar messages [ 6020.603124] Lustre: server umount lustre-MDT0000 complete [ 6024.375333] LustreError: 144601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332841 with bad export cookie 4027502309225923400 [ 6024.383968] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6024.388384] LustreError: 144601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6024.408457] LustreError: 147869:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000884ea89f x1866814124503680/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6024.425811] LustreError: 147869:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x20000000a:0x0:0x0]: rc = -5 [ 6024.434260] LustreError: 147869:0:(llog_cat.c:443:llog_cat_close()) lustre-MDT0000-osp-MDT0001: failure destroying log during cleanup: rc = -5 [ 6024.811871] Lustre: server umount lustre-MDT0001 complete [ 6040.205548] Lustre: server umount lustre-OST0000 complete [ 6065.497893] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 6080.000657] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6128.150337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6128.702863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6128.839141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6128.976352] Lustre: lustre-MDT0000: new disk, initializing [ 6129.130961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6129.181872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6133.717305] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6147.101665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6147.281242] Lustre: 151023:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6147.333499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6147.336801] Lustre: Skipped 1 previous similar message [ 6147.430501] Lustre: lustre-MDT0001: new disk, initializing [ 6147.524139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6147.569506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6147.588545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6152.703736] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6162.233048] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6167.656588] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6175.131141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6175.444895] Lustre: lustre-OST0000: new disk, initializing [ 6175.452736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6175.462366] Lustre: 152769:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6175.572169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.505730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6181.517231] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6181.694065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6183.083304] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6192.605634] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6198.459147] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6198.647031] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6204.010297] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6204.266481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6207.460388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6207.469636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6207.479825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6207.489467] Lustre: Skipped 1 previous similar message [ 6207.512197] Lustre: Skipped 1 previous similar message [ 6212.599490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6217.701757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6217.709044] Lustre: Skipped 2 previous similar messages [ 6221.279296] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6221.483947] Lustre: server umount lustre-OST0000 complete [ 6227.943217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6227.956178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6227.972849] Lustre: Skipped 1 previous similar message [ 6227.994755] Lustre: Skipped 1 previous similar message [ 6240.223131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6240.600523] Lustre: server umount lustre-MDT0000 complete [ 6243.296836] LustreError: 152410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6243.329579] LustreError: 152410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6245.498457] LustreError: 151015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333062 with bad export cookie 4027502309225924401 [ 6245.502168] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6245.513678] LustreError: 151015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6246.209617] Lustre: server umount lustre-MDT0001 complete [ 6251.153345] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 12:57:45 (1780333065) [ 6253.549494] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6256.187659] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 12:57:50 (1780333070) [ 6266.175337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6266.763367] LustreError: 155039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6266.877631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6271.976622] LustreError: 155040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6272.408830] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6277.095743] LustreError: 155039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6282.214540] LustreError: 155040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6283.403866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6290.496596] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6299.211572] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6306.078638] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6318.170622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6318.772834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6318.781766] Lustre: Skipped 1 previous similar message [ 6326.974412] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6335.065169] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6348.776065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6348.783316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6348.786229] Lustre: Skipped 1 previous similar message [ 6348.793305] Lustre: Skipped 6 previous similar messages [ 6353.242433] Lustre: server umount lustre-MDT0000 complete [ 6353.904830] LustreError: 155764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6353.927346] LustreError: 155764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6358.214234] LustreError: 155020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333174 with bad export cookie 4027502309225925325 [ 6358.222855] LustreError: 155020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6358.226121] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6359.023783] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6359.045158] Lustre: Skipped 1 previous similar message [ 6365.067788] Lustre: server umount lustre-MDT0001 complete [ 6372.655319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6388.205430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6388.633468] LustreError: 158572:0:(llog.c:1646:llog_backup()) MGC192.168.202.120@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6388.640586] Lustre: 158572:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.120@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6391.371782] LustreError: 158577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6391.405591] LustreError: 158577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6423.525288] LustreError: 158577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6423.541604] LustreError: 158577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 6449.120080] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6449.125072] Lustre: 158572:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.120@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6488.545330] LustreError: 158578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6488.560881] LustreError: 158578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 43 previous similar messages [ 6490.107738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6495.209359] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6495.444666] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6504.985932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6512.611174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6512.625157] LustreError: Skipped 1 previous similar message [ 6533.168389] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6533.619673] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6539.836936] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6541.998069] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6547.199520] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6549.352494] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6572.511980] Lustre: server umount lustre-MDT0000 complete [ 6574.560053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6574.574181] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.621968] Lustre: Skipped 1 previous similar message [ 6577.292826] LustreError: 155019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333393 with bad export cookie 4027502309225931814 [ 6577.304441] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6577.315193] LustreError: 155019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6578.061892] Lustre: server umount lustre-MDT0001 complete [ 6594.789957] Lustre: server umount lustre-OST0000 complete [ 6625.459427] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 6645.567112] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6690.888096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6691.196404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6691.244879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6691.329436] Lustre: lustre-MDT0000: new disk, initializing [ 6691.488734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6691.495273] Lustre: Skipped 1 previous similar message [ 6691.508442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6696.818969] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6708.167974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6708.346596] Lustre: 163457:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6708.706200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6713.078442] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6721.631871] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6726.002507] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6733.651147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6733.989279] Lustre: lustre-OST0000: new disk, initializing [ 6733.991556] Lustre: Skipped 1 previous similar message [ 6734.000108] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6734.002534] Lustre: Skipped 2 previous similar messages [ 6734.010916] Lustre: 165200:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6742.212644] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6742.592330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6742.604571] Lustre: Skipped 1 previous similar message [ 6742.614204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6742.773329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6751.973626] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6757.468744] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6757.684381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6762.642842] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6763.005962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6768.616705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6768.624318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6768.646393] Lustre: Skipped 1 previous similar message [ 6768.651937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6768.679062] Lustre: Skipped 5 previous similar messages [ 6771.894324] Lustre: server umount lustre-OST0000 complete [ 6782.472390] Lustre: server umount lustre-MDT0000 complete [ 6783.973100] LustreError: 163466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6783.992512] LustreError: 163466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 40 previous similar messages [ 6787.131536] LustreError: 163451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333603 with bad export cookie 4027502309225948600 [ 6787.139924] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6787.146106] LustreError: 163451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6787.627738] Lustre: server umount lustre-MDT0001 complete [ 6798.841171] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 13:06:53 (1780333613) [ 6806.984261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6812.747074] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6824.054241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6830.134647] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6837.845934] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6843.388671] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6851.786635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6860.820417] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6868.517814] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6885.703938] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 13:08:20 (1780333700) [ 6894.561847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6894.571276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6894.587495] Lustre: Skipped 3 previous similar messages [ 6894.602926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6894.617163] Lustre: Skipped 2 previous similar messages [ 6897.779121] Lustre: server umount lustre-OST0000 complete [ 6908.878324] Lustre: server umount lustre-MDT0000 complete [ 6913.320173] LustreError: 167447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333729 with bad export cookie 4027502309225949517 [ 6913.329265] LustreError: 167447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6913.329816] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6913.664060] Lustre: server umount lustre-MDT0001 complete [ 6927.327779] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 6930.459693] Key type lgssc unregistered [ 6930.725358] LNet: 170844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6930.730037] LNetError: 170844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6930.764474] LNet: Removed LNI 192.168.202.120@tcp [ 6931.860856] Key type .llcrypt unregistered [ 6931.863780] Key type ._llcrypt unregistered [ 6955.872470] Key type ._llcrypt registered [ 6955.875229] Key type .llcrypt registered [ 6955.985822] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6957.064865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6957.226776] alg: No test for adler32 (adler32-zlib) [ 6958.344333] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 6958.656947] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 6960.365976] Key type lgssc registered [ 6961.784923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6972.156794] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6972.189650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6987.871340] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6992.992223] LustreError: 172070:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [ 6998.175711] LustreError: 172070:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6998.197948] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6998.207093] LustreError: MGC192.168.202.120@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6998.223103] LustreError: 172070:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6998.238630] LustreError: 172070:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -22 [ 6998.259650] LustreError: 172070:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 6998.423740] Lustre: server umount lustre-OST0000 complete [ 6998.425713] LustreError: 172070:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7007.682585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7009.304824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7014.684140] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7018.481305] LustreError: 172571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7018.501741] LustreError: 172571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7023.586202] LustreError: 172570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7024.403992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.771738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7029.381768] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7035.799786] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7040.719645] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7049.137418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7049.438091] LustreError: 174165:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7049.455296] Lustre: 174165:0:(mgc_request_server.c:748:mgc_llog_local_copy()) MGC192.168.202.120@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7049.713735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7056.684723] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7064.175354] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7069.153611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7069.163636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7069.171777] Lustre: Skipped 2 previous similar messages [ 7069.181364] Lustre: Skipped 2 previous similar messages [ 7072.970515] Lustre: server umount lustre-MDT0000 complete [ 7074.273922] LustreError: 173284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7074.293301] LustreError: 173284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7077.300372] LustreError: 172550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333893 with bad export cookie 7064814918873493975 [ 7077.311508] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7077.999784] Lustre: server umount lustre-MDT0001 complete [ 7093.267888] Lustre: server umount lustre-OST0000 complete [ 7104.637862] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 7116.701675] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7158.908346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7159.190760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7159.256538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7159.404139] Lustre: lustre-MDT0000: new disk, initializing [ 7159.533121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7159.560487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7164.309730] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7175.861358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7176.007371] Lustre: 177543:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7176.083520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7176.088136] Lustre: Skipped 1 previous similar message [ 7176.184296] Lustre: lustre-MDT0001: new disk, initializing [ 7176.295315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7176.347368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7176.367403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7181.064363] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7189.181690] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7193.313189] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7200.941695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7201.220977] Lustre: lustre-OST0000: new disk, initializing [ 7201.224405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7201.229643] Lustre: 179287:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7201.310630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7208.075542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7208.101627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7208.239090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7208.764951] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7218.296151] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7223.366843] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7223.592095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7228.930208] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7229.163933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7234.020895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7234.033604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.047396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7234.056585] Lustre: Skipped 1 previous similar message [ 7237.886788] Lustre: server umount lustre-OST0000 complete [ 7242.727587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7242.740292] LustreError: Skipped 1 previous similar message [ 7242.753209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7242.776858] Lustre: Skipped 1 previous similar message [ 7242.786669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7248.551769] Lustre: server umount lustre-MDT0000 complete [ 7249.376640] LustreError: 178655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7249.395818] LustreError: 178655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7253.117374] LustreError: 177537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334069 with bad export cookie 7064814918873494696 [ 7253.124425] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7253.128706] LustreError: 177537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7253.781212] Lustre: server umount lustre-MDT0001 complete [ 7265.149156] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 13:14:39 (1780334079) [ 7289.833486] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 7292.729443] Key type lgssc unregistered [ 7293.028781] LNet: 182072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7293.035742] LNetError: 182072:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7293.061866] LNet: Removed LNI 192.168.202.120@tcp [ 7293.890464] Key type .llcrypt unregistered [ 7293.894184] Key type ._llcrypt unregistered [ 7314.506292] Key type ._llcrypt registered [ 7314.509711] Key type .llcrypt registered [ 7314.633145] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7315.966604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7316.099513] alg: No test for adler32 (adler32-zlib) [ 7317.262634] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 7317.515098] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 7319.199328] Key type lgssc registered [ 7320.580535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7329.380089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7329.405813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7331.124900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7335.820296] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7340.538932] LustreError: 183275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7340.572792] LustreError: 183275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7345.284411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7345.635072] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7345.971576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7350.558769] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7357.994969] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7363.426696] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7373.837867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7374.152994] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7374.156610] LustreError: 184916:0:(llog.c:1687:llog_backup()) MGC192.168.202.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7374.162280] Lustre: 184916:0:(mgc_request_server.c:745:mgc_llog_local_copy()) MGC192.168.202.120@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7374.302822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7381.459150] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7388.901544] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7392.224652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7392.231231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7392.265249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7392.741861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7392.797602] Lustre: Skipped 1 previous similar message [ 7393.827360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7393.846634] Lustre: Skipped 4 previous similar messages [ 7397.869461] Lustre: server umount lustre-MDT0000 complete [ 7398.896714] LustreError: 184246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7398.932669] LustreError: 184246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7402.042803] LustreError: 183256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334218 with bad export cookie 4871101687821574268 [ 7402.060267] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7402.079970] LustreError: 183988:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006e9f1c69 x1866815644923648/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7402.542475] Lustre: server umount lustre-MDT0001 complete [ 7417.238551] Lustre: server umount lustre-OST0000 complete [ 7428.553424] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 7441.920766] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7489.360327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7489.616885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7489.653787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7489.760544] Lustre: lustre-MDT0000: new disk, initializing [ 7489.915725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7489.959652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7494.870718] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7506.810043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7506.981943] Lustre: 188298:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7507.048172] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7507.050579] Lustre: Skipped 1 previous similar message [ 7507.053456] LustreError: 189064:0:(llog.c:1687:llog_backup()) MGC192.168.202.120@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7507.059781] LustreError: 189064:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7507.065499] Lustre: 189064:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.120@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7507.088969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7507.093103] Lustre: Skipped 1 previous similar message [ 7507.248288] Lustre: lustre-MDT0001: new disk, initializing [ 7507.396680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7507.444952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7507.458316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7511.746308] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7519.888724] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7523.324844] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7530.245556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7530.519406] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7530.523260] Lustre: Skipped 1 previous similar message [ 7530.526371] LustreError: 190017:0:(llog.c:1687:llog_backup()) MGC192.168.202.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7530.534412] LustreError: 190017:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7530.538558] Lustre: 190017:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.120@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7530.549905] Lustre: 190017:0:(mgc_request_server.c:768:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7530.575983] Lustre: lustre-OST0000: new disk, initializing [ 7530.579826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7530.584506] Lustre: 190041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7530.706852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7536.995606] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7540.894173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7540.914286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7540.982858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7546.012206] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7551.640913] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7552.018450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7558.148472] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7558.365799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7561.721954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7561.726050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7561.736218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7561.740679] Lustre: Skipped 1 previous similar message [ 7561.757793] Lustre: Skipped 1 previous similar message [ 7566.825430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7566.836995] Lustre: Skipped 1 previous similar message [ 7567.150295] Lustre: server umount lustre-OST0000 complete [ 7573.999242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7574.007413] LustreError: Skipped 1 previous similar message [ 7574.016826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7577.056292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7577.059748] Lustre: Skipped 2 previous similar messages [ 7578.811455] Lustre: server umount lustre-MDT0000 complete [ 7582.189168] LustreError: 188303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7582.216186] LustreError: 188303:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7583.382036] LustreError: 188290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334400 with bad export cookie 4871101687821574982 [ 7583.406598] LustreError: 188290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7583.411456] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7583.845242] Lustre: server umount lustre-MDT0001 complete [ 7593.196471] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 13:20:08 (1780334408) [ 7612.685811] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7624.193421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7624.668126] LustreError: 193005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7624.758875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7629.800563] LustreError: 193006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7630.297333] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7639.106193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.298360] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7648.660364] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7656.425344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7656.839835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7656.846493] Lustre: Skipped 1 previous similar message [ 7664.210284] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7674.539158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7674.723455] Lustre: lustre-OST0001: new disk, initializing [ 7674.729596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7674.752761] Lustre: 195265:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7680.686514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7680.694585] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7680.852704] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7682.934785] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7696.457736] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7706.780426] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7709.571312] Lustre: DEBUG MARKER: ADD OST3 [ 7716.905588] loop: module loaded [ 7717.003752] loop0: detected capacity change from 0 to 204800000 [ 7717.267033] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7717.297724] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7717.311108] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7725.805667] loop0: detected capacity change from 0 to 204800000 [ 7725.811876] Lustre: DEBUG MARKER: STOP OST3 [ 7732.590266] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7732.863859] Lustre: lustre-OST0003: new disk, initializing [ 7732.868738] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7732.884951] Lustre: 197290:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7733.051464] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7733.060332] Lustre: Skipped 1 previous similar message [ 7734.399589] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7734.563695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7734.577254] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7734.617693] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7736.369707] Lustre: Failing over lustre-OST0003 [ 7736.419204] LustreError: 197419:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7736.666133] Lustre: server umount lustre-OST0003 complete [ 7739.878885] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7739.895275] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7739.913674] LustreError: 194459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7739.919749] Lustre: Skipped 2 previous similar messages [ 7739.940356] LustreError: 194459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7740.401758] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7750.889617] Lustre: DEBUG MARKER: START OST3 again [ 7759.517045] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7759.963339] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7760.402355] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7761.184526] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7761.186081] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7767.042729] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7792.150714] Lustre: Failing over lustre-OST0003 [ 7792.247893] Lustre: server umount lustre-OST0003 complete [ 7795.702745] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7795.712970] LustreError: 194460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7795.726817] Lustre: Skipped 1 previous similar message [ 7795.742815] LustreError: 194460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 7813.604667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7813.621972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.655647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7818.764511] Lustre: server umount lustre-MDT0000 complete [ 7823.285983] LustreError: 192985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334639 with bad export cookie 4871101687821575913 [ 7823.292197] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7823.300586] LustreError: 192985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7828.450956] LustreError: 193000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7828.460866] LustreError: 193000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 7829.661593] Lustre: server umount lustre-MDT0001 complete [ 7834.315787] Lustre: server umount lustre-OST0000 complete [ 7839.234144] Lustre: server umount lustre-OST0001 complete [ 7848.748259] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 7861.557300] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7909.840723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7910.161710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7910.208298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7910.300780] Lustre: lustre-MDT0000: new disk, initializing [ 7910.463628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.477561] Lustre: Skipped 1 previous similar message [ 7910.509366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7915.955792] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7928.410601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7928.555375] Lustre: 202649:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7928.632522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7928.637982] Lustre: Skipped 1 previous similar message [ 7928.734651] Lustre: lustre-MDT0001: new disk, initializing [ 7928.958754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7928.971149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7934.228738] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7942.755803] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7947.324243] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7954.966532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7955.314654] Lustre: 204393:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7957.028536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7957.142460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7963.105368] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7971.304372] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7977.211509] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7977.457483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7983.100686] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7983.356682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7987.687401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7987.703594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7987.705809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7987.724100] Lustre: Skipped 6 previous similar messages [ 7987.738133] Lustre: Skipped 9 previous similar messages [ 7991.772982] Lustre: server umount lustre-OST0000 complete [ 8002.766444] Lustre: server umount lustre-MDT0000 complete [ 8003.050254] LustreError: 203884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8003.060062] LustreError: 203884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8007.312926] LustreError: 202641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334823 with bad export cookie 4871101687821578195 [ 8007.352925] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8007.616176] Lustre: server umount lustre-MDT0001 complete [ 8012.174056] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 13:27:07 (1780334827) [ 8030.878629] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 8041.985158] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8085.591969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8085.852934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8085.946334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8085.950594] Lustre: Skipped 1 previous similar message [ 8086.072649] Lustre: lustre-MDT0000: new disk, initializing [ 8086.075824] Lustre: Skipped 1 previous similar message [ 8086.232404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8086.241522] Lustre: Skipped 1 previous similar message [ 8092.787768] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8106.969230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8107.104816] Lustre: 208821:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8107.401987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8113.123959] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8122.353777] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8127.819518] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8135.548168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8135.973698] Lustre: 210569:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8137.978671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8144.037608] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8152.825420] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8158.890916] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8159.147375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8165.356950] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8165.704783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8168.930183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8168.932664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8168.942745] LustreError: Skipped 1 previous similar message [ 8168.943800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8168.943805] Lustre: Skipped 2 previous similar messages [ 8169.013682] Lustre: Skipped 3 previous similar messages [ 8182.239838] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8182.417990] Lustre: server umount lustre-OST0000 complete [ 8193.413215] Lustre: server umount lustre-MDT0000 complete [ 8197.610348] LustreError: 208834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8197.643041] LustreError: 208834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8198.510887] LustreError: 208814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335015 with bad export cookie 4871101687821579126 [ 8198.516390] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8198.527355] LustreError: 208814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8199.289536] Lustre: server umount lustre-MDT0001 complete [ 8209.728041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8210.518752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8210.531777] Lustre: Skipped 5 previous similar messages [ 8215.682482] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8227.853817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8235.789698] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8243.749566] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8249.511489] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8257.064064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8264.321409] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8271.016193] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8294.368915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8295.581340] Lustre: server umount lustre-OST0000 complete [ 8301.551115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8301.556288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8301.559220] Lustre: Skipped 5 previous similar messages [ 8301.573433] Lustre: Skipped 14 previous similar messages [ 8315.359515] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8315.633231] Lustre: server umount lustre-MDT0000 complete [ 8319.824859] LustreError: 212623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335136 with bad export cookie 4871101687821580043 [ 8319.835569] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8320.265822] Lustre: server umount lustre-MDT0001 complete [ 8411.186542] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 8414.405541] Key type lgssc unregistered [ 8414.713960] LNet: 215975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8414.724717] LNetError: 215975:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8414.739208] LNet: Removed LNI 192.168.202.120@tcp [ 8415.919566] Key type .llcrypt unregistered [ 8415.930957] Key type ._llcrypt unregistered [ 8448.628366] Key type ._llcrypt registered [ 8448.632587] Key type .llcrypt registered [ 8448.796199] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 8468.582959] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8470.032749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8470.295232] alg: No test for adler32 (adler32-zlib) [ 8471.368563] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 8471.681400] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 8473.351182] Key type lgssc registered [ 8474.688606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8515.096604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8515.146196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8516.553191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8516.622976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8516.725955] Lustre: lustre-MDT0000: new disk, initializing [ 8516.916207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8516.980503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8523.119924] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8536.520122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8536.672303] Lustre: 219297:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8536.727266] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8536.731466] Lustre: Skipped 1 previous similar message [ 8536.821081] Lustre: lustre-MDT0001: new disk, initializing [ 8536.977289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8537.031477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8537.043887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8541.644787] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8549.729764] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8554.112926] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8561.970431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8562.290319] Lustre: lustre-OST0000: new disk, initializing [ 8562.297660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8562.309684] Lustre: 221043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8562.456524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8568.495304] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8570.432173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8570.444784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8570.624895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8576.654745] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8582.353936] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8582.726406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8587.807711] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8587.989759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8591.333387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8591.342322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8591.345758] LustreError: Skipped 1 previous similar message [ 8591.354956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8591.367382] Lustre: Skipped 1 previous similar message [ 8591.394844] Lustre: Skipped 1 previous similar message [ 8596.486576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8596.489698] Lustre: Skipped 1 previous similar message [ 8596.757981] Lustre: server umount lustre-OST0000 complete [ 8603.624144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8603.654321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8603.699536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8606.709487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8606.719360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8608.560753] Lustre: server umount lustre-MDT0000 complete [ 8612.249488] LustreError: 219290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335428 with bad export cookie 5922869074423781002 [ 8612.256729] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8612.259943] LustreError: 219290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8612.583316] Lustre: server umount lustre-MDT0001 complete [ 8633.953830] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8645.165143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8645.760770] LustreError: 223814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8645.782125] LustreError: 223814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8645.833332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8650.417713] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8651.234601] LustreError: 223815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8656.360393] LustreError: 223814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8660.415816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8660.647703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8665.983826] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8669.302975] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8676.858439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8677.166298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8683.651882] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8694.685322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8694.878749] Lustre: lustre-OST0001: new disk, initializing [ 8694.883200] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8694.888431] Lustre: 226072:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8694.988317] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8702.311851] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8704.151127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8704.164191] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8704.243152] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8715.231762] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8726.236944] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8739.739331] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 13:39:14 (1780335554) [ 8742.881160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8742.888078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8742.915889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8742.922076] Lustre: Skipped 1 previous similar message [ 8755.687098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8755.696418] Lustre: Skipped 11 previous similar messages [ 8756.191386] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8756.357718] Lustre: server umount lustre-MDT0000 complete [ 8759.741761] LustreError: 223795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335576 with bad export cookie 5922869074423781926 [ 8759.745022] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8759.750894] LustreError: 223795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8759.981819] Lustre: server umount lustre-MDT0001 complete [ 8774.355817] Lustre: server umount lustre-OST0000 complete [ 8777.184089] Lustre: 217523:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780335577/real 1780335577] req@ffff897545f44a80 x1866816855070336/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780335593 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8777.217933] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8777.236911] Lustre: Skipped 3 previous similar messages [ 8778.553814] Lustre: server umount lustre-OST0001 complete [ 8787.155172] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 8799.152277] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8831.623521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8831.980334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8832.033900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8832.122414] Lustre: lustre-MDT0000: new disk, initializing [ 8832.265952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8832.315717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8836.913592] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8848.173285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8848.357436] Lustre: 229967:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8848.462247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8848.467373] Lustre: Skipped 1 previous similar message [ 8848.572180] Lustre: lustre-MDT0001: new disk, initializing [ 8848.741666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8848.766172] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8854.435452] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8863.290579] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8868.378276] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8876.883345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8877.128987] Lustre: lustre-OST0000: new disk, initializing [ 8877.133466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8877.143427] Lustre: 231711:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8879.031154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8879.048371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8879.279826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8884.986731] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8895.050409] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8899.971914] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8900.257159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8905.625946] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8905.886508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8909.801279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8909.803264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8909.820421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8909.833159] Lustre: Skipped 1 previous similar message [ 8909.851819] Lustre: Skipped 1 previous similar message [ 8914.259096] Lustre: server umount lustre-OST0000 complete [ 8933.343153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8933.589799] Lustre: server umount lustre-MDT0000 complete [ 8935.397320] LustreError: 229975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8935.412239] LustreError: 229975:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8937.441852] LustreError: 229960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335754 with bad export cookie 5922869074423784068 [ 8937.458227] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8938.029488] Lustre: server umount lustre-MDT0001 complete [ 8947.353046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8948.239723] LustreError: 233785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.376305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8948.379490] Lustre: Skipped 2 previous similar messages [ 8954.214733] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8957.920338] LustreError: 233786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8957.944090] LustreError: 233786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8965.234804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8971.439464] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8978.038205] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8982.041385] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8989.175044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8995.998617] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9002.280548] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9151.461845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9151.465265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9151.466305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9151.466329] Lustre: Skipped 6 previous similar messages [ 9151.494607] Lustre: Skipped 3 previous similar messages [ 9164.257140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9164.442715] Lustre: server umount lustre-OST0000 complete [ 9170.407614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9183.199488] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9183.542274] Lustre: server umount lustre-MDT0000 complete [ 9187.308944] LustreError: 233786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9187.324451] LustreError: 233786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9188.180852] LustreError: 233767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336004 with bad export cookie 5922869074423784992 [ 9188.185100] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9188.190059] LustreError: 233767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9188.493532] Lustre: server umount lustre-MDT0001 complete [ 9269.937992] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 9273.107821] Key type lgssc unregistered [ 9273.450956] LNet: 237035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9273.459358] LNetError: 237035:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9273.491243] LNet: Removed LNI 192.168.202.120@tcp [ 9274.303180] Key type .llcrypt unregistered [ 9274.307077] Key type ._llcrypt unregistered [ 9305.717589] Key type ._llcrypt registered [ 9305.721921] Key type .llcrypt registered [ 9305.938907] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 9326.884642] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9329.030052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9329.059073] alg: No test for adler32 (adler32-zlib) [ 9330.663156] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 9331.144514] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 9332.975309] Key type lgssc registered [ 9334.431212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9383.273530] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9394.604164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9394.621717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9395.903057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9395.953702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9396.061891] Lustre: lustre-MDT0000: new disk, initializing [ 9396.164106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9396.183540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9401.102625] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9412.630124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9412.835952] Lustre: 241061:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9412.930945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9412.938113] Lustre: Skipped 1 previous similar message [ 9413.095251] Lustre: lustre-MDT0001: new disk, initializing [ 9413.214979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9413.267500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9413.272897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9418.238471] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9424.216106] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9432.385687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9433.033864] Lustre: lustre-OST0000: new disk, initializing [ 9433.044775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9433.072951] Lustre: 242661:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9433.197125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9439.875151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9439.889753] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9440.052071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9441.697700] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9456.878232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9457.162952] Lustre: lustre-OST0001: new disk, initializing [ 9457.167344] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9457.176094] Lustre: 243514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9457.369774] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9463.461867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9463.482771] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9463.595446] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9464.667902] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9477.204063] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9488.174786] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9504.740837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9504.746464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9504.759569] Lustre: Skipped 3 previous similar messages [ 9504.774862] Lustre: Skipped 3 previous similar messages [ 9506.872042] Lustre: server umount lustre-MDT0000 complete [ 9511.036062] LustreError: 241051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336327 with bad export cookie 13268599411793884912 [ 9511.048370] LustreError: 241051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9511.053384] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9511.775545] Lustre: server umount lustre-MDT0001 complete [ 9527.065701] Lustre: server umount lustre-OST0000 complete [ 9531.107712] Lustre: 238586:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780336331/real 1780336331] req@ffff897648232680 x1866817755764480/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780336347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9531.150504] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9531.754120] Lustre: server umount lustre-OST0001 complete [ 9546.870779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9557.203637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9566.119187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9577.677788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9601.815351] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9613.514258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9613.532135] Lustre: lustre-MDT0000: reset Object Index mappings [ 9614.293054] LustreError: 248494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9614.331826] LustreError: 248494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9614.481745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9619.943187] LustreError: 248495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9620.431796] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9624.050202] LustreError: 248494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9630.177939] LustreError: 248495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9630.202045] LustreError: 248495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9630.258094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9630.278864] Lustre: lustre-MDT0001: reset Object Index mappings [ 9630.696903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9635.987786] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9639.269971] Lustre: 249606:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9645.764942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9654.452617] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9656.375460] LustreError: 249962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9658.481660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9664.631986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9664.902166] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9664.915799] Lustre: Skipped 1 previous similar message [ 9670.153848] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9672.946679] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9682.642088] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9686.669520] Lustre: 251442:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9692.824270] Lustre: server umount lustre-MDT0000 complete [ 9695.721732] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9695.726407] LustreError: 249577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9695.733211] Lustre: Skipped 3 previous similar messages [ 9695.745311] LustreError: 249577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9697.356883] LustreError: 248476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336513 with bad export cookie 13268599411793891744 [ 9697.360771] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9697.377659] LustreError: 248476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9697.878962] Lustre: server umount lustre-MDT0001 complete [ 9711.709321] Lustre: server umount lustre-OST0000 complete [ 9726.149727] Lustre: server umount lustre-OST0001 complete [ 9749.084176] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 13:56:04 (1780336564) [ 9769.167888] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 9780.152271] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9824.249732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9824.520569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9824.554241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9824.667992] Lustre: lustre-MDT0000: new disk, initializing [ 9824.797768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9824.819827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9830.468795] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9842.038578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9842.114251] Lustre: 255664:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9842.167715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9842.172058] Lustre: Skipped 1 previous similar message [ 9842.238544] Lustre: lustre-MDT0001: new disk, initializing [ 9842.360438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9842.376460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9847.959854] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9857.119429] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9862.209278] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9870.132700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9870.485423] Lustre: lustre-OST0000: new disk, initializing [ 9870.496432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9870.506765] Lustre: 257409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9872.671400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9872.688233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9872.798937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9876.603614] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9883.830230] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9888.173427] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9888.400710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9892.896320] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9893.112544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9898.504155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9898.507316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9898.507808] LustreError: Skipped 1 previous similar message [ 9898.518940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9898.534287] Lustre: Skipped 1 previous similar message [ 9898.554284] Lustre: Skipped 1 previous similar message [ 9901.664255] Lustre: server umount lustre-OST0000 complete [ 9908.708272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9908.724520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9908.733098] Lustre: Skipped 1 previous similar message [ 9908.746975] Lustre: Skipped 1 previous similar message [ 9913.022632] Lustre: server umount lustre-MDT0000 complete [ 9913.837168] LustreError: 255676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9913.869986] LustreError: 255676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9917.179372] LustreError: 255656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336733 with bad export cookie 13268599411793893830 [ 9917.181986] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9917.195865] LustreError: 255656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9917.504725] Lustre: server umount lustre-MDT0001 complete [ 9940.338871] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9954.891174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9955.271895] LustreError: 260181:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9955.325975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9955.338535] Lustre: Skipped 2 previous similar messages [ 9962.909667] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9974.449543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9980.547330] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9984.862736] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9995.580502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10004.902230] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10015.439230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10015.628869] Lustre: lustre-OST0001: new disk, initializing [10015.635629] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10015.642718] Lustre: 262444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10017.624610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10017.634161] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10017.757083] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10022.412186] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10033.085490] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10036.705071] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10042.214970] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10042.232767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10048.487747] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10048.504460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10048.504673] Lustre: Skipped 2 previous similar messages [10048.517909] Lustre: Skipped 3 previous similar messages [10053.604334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10054.722471] Lustre: server umount lustre-MDT0000 complete [10058.190536] LustreError: 262443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336874 with bad export cookie 13268599411793894761 [10058.197612] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10058.212689] LustreError: 262443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10058.643744] Lustre: server umount lustre-MDT0001 complete [10072.644642] Lustre: server umount lustre-OST0000 complete [10075.103176] Lustre: 238585:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780336875/real 1780336875] req@ffff897681cf2a00 x1866817755934208/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780336891 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10075.144094] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10075.162535] Lustre: Skipped 1 previous similar message [10077.518899] Lustre: server umount lustre-OST0001 complete [10096.934323] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [10110.546758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10111.066682] LustreError: 265473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10111.096170] LustreError: 265473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10111.161284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10111.176333] Lustre: Skipped 3 previous similar messages [10117.959438] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10128.158041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10133.563233] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10137.237483] Lustre: 266582:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10144.929331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10151.448695] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10155.691643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10160.835319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10166.292170] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10167.745615] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10176.286467] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10181.133482] Lustre: 268428:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10187.759133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10187.868760] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10199.334233] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 14:03:34 (1780337014) [10215.393491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10215.409706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10215.427633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10215.441449] Lustre: Skipped 3 previous similar messages [10221.145225] Lustre: server umount lustre-MDT0000 complete [10226.066530] LustreError: 266584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337042 with bad export cookie 13268599411793901285 [10226.084148] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10231.713554] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10231.723977] Lustre: Skipped 7 previous similar messages [10232.743770] Lustre: server umount lustre-MDT0001 complete [10244.410790] Lustre: server umount lustre-OST0000 complete [10248.694632] Lustre: server umount lustre-OST0001 complete [10256.137830] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [10273.968899] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [10329.189110] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [10340.965197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10341.247659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10341.271835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10341.344930] Lustre: lustre-MDT0000: new disk, initializing [10341.472391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10347.369447] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10358.797751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10358.908870] Lustre: 272642:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10359.241398] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10363.745466] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10376.804770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10377.049110] Lustre: 274241:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10377.143989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10377.153336] Lustre: Skipped 5 previous similar messages [10378.782105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10384.020884] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10395.229321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10395.360573] Lustre: 275096:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10396.791572] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10396.801769] Lustre: Skipped 1 previous similar message [10396.963611] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10402.012725] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10411.780099] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10415.053488] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10415.059808] Lustre: Skipped 1 previous similar message [10432.997681] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 14:07:28 (1780337248) [10436.065030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10436.071087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10436.093314] Lustre: Skipped 5 previous similar messages [10436.115504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10437.683581] Lustre: server umount lustre-MDT0000 complete [10438.116707] LustreError: 272652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10438.138416] LustreError: 272652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [10442.057823] LustreError: 275094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337258 with bad export cookie 13268599411793908957 [10442.061301] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10442.077905] LustreError: 275094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10442.445729] Lustre: server umount lustre-MDT0001 complete [10456.955149] Lustre: server umount lustre-OST0000 complete [10459.104281] Lustre: 238583:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780337259/real 1780337259] req@ffff897542ef9880 x1866817756092416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780337275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10460.778821] Lustre: server umount lustre-OST0001 complete [10469.773257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10474.630255] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10483.184969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10488.086677] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10494.627470] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10499.434608] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10507.179292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10514.571119] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10520.639996] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10539.423260] loop1: detected capacity change from 0 to 204800000 [10539.731983] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10539.745387] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10539.757615] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10545.538213] loop1: detected capacity change from 0 to 204800000 [10550.898212] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10550.968247] LustreError: 277316:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs9532: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10550.986780] LustreError: 277316:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs9532 [10550.994470] LustreError: 277316:0:(mgs_handler.c:517:mgs_target_reg()) fs9532-MDT0001: Error checking target index: rc = -13 [10551.003578] LustreError: fs9532-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10551.010192] LustreError: 280756:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -13 [10551.016587] LustreError: 280756:0:(tgt_mount.c:2084:server_put_super()) no obd fs9532-MDT0001 [10551.022190] LustreError: 280756:0:(tgt_mount.c:116:server_deregister_mount()) fs9532-MDT0001 not registered [10551.072558] Lustre: server umount fs9532-MDT0001 complete [10551.077391] LustreError: 280756:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10559.260663] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10559.332446] Lustre: Setting parameter fs9532-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs9532-MDT0001 [10559.360099] Lustre: srv-fs9532-MDT0001: No data found on store. Initialize space. [10559.367116] Lustre: Skipped 4 previous similar messages [10559.467233] Lustre: fs9532-MDT0001: new disk, initializing [10559.472409] Lustre: Skipped 3 previous similar messages [10559.590415] Lustre: srv-fs9532-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10559.596523] Lustre: 281224:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9532-MDT0001-mdtlov: can't create new llog: rc = -115 [10559.615304] LustreError: 281224:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9532-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10559.635672] LustreError: 281226:0:(fld_handler.c:275:fld_server_lookup()) srv-fs9532-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [10559.646393] LustreError: 281226:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) fs9532-MDT0001: Trash dir setup failed: rc = -5 [10560.136087] LustreError: 281226:0:(fld_handler.c:275:fld_server_lookup()) srv-fs9532-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [10560.148216] LustreError: 281226:0:(fld_handler.c:275:fld_server_lookup()) Skipped 2314 previous similar messages [10560.152367] LustreError: 281226:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) fs9532-MDT0001: Trash dir setup failed: rc = -5 [10560.170760] LustreError: 281226:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 2314 previous similar messages [10561.136245] LustreError: 281226:0:(fld_handler.c:275:fld_server_lookup()) srv-fs9532-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [10561.148700] LustreError: 281226:0:(fld_handler.c:275:fld_server_lookup()) Skipped 4397 previous similar messages [10561.155288] LustreError: 281226:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) fs9532-MDT0001: Trash dir setup failed: rc = -5 [10561.161785] LustreError: 281226:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 4397 previous similar messages [10563.143187] LustreError: 281226:0:(fld_handler.c:275:fld_server_lookup()) srv-fs9532-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [10563.152328] LustreError: 281226:0:(fld_handler.c:275:fld_server_lookup()) Skipped 10226 previous similar messages [10563.163244] LustreError: 281226:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) fs9532-MDT0001: Trash dir setup failed: rc = -5 [10563.177437] LustreError: 281226:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 10226 previous similar messages [10564.619302] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10567.142096] LustreError: 281226:0:(fld_handler.c:275:fld_server_lookup()) srv-fs9532-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [10567.148146] LustreError: 281226:0:(fld_handler.c:275:fld_server_lookup()) Skipped 23149 previous similar messages [10567.160076] LustreError: 281226:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) fs9532-MDT0001: Trash dir setup failed: rc = -5 [10567.172601] LustreError: 281226:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 23229 previous similar messages [10571.965720] Lustre: server umount fs9532-MDT0001 complete [10583.845452] loop1: detected capacity change from 0 to 204800000 [10589.269479] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10589.489147] Lustre: 282778:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9532-MDT0001-mdtlov: can't create new llog: rc = -115 [10589.493991] LustreError: 282778:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9532-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10589.513990] LustreError: 282780:0:(fld_handler.c:275:fld_server_lookup()) srv-fs9532-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [10589.524849] LustreError: 282780:0:(fld_handler.c:275:fld_server_lookup()) Skipped 24893 previous similar messages [10589.534966] LustreError: 282780:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) fs9532-MDT0001: Trash dir setup failed: rc = -5 [10589.544460] LustreError: 282780:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 24813 previous similar messages [10591.719367] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10595.447460] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10596.839431] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10600.831522] Lustre: server umount fs9532-MDT0001 complete [10611.685118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10611.702352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10611.730603] Lustre: Skipped 4 previous similar messages [10611.737523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10616.937724] Lustre: server umount lustre-MDT0000 complete [10627.887140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10628.039814] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10633.706916] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [10633.719476] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [10633.741533] Lustre: Skipped 2 previous similar messages [10634.300919] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10642.679967] loop1: detected capacity change from 0 to 204800000 [10648.594991] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10648.928125] Lustre: 285191:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9532-MDT0001-mdtlov: can't create new llog: rc = -115 [10648.939207] LustreError: 285191:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9532-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10648.968373] LustreError: 285193:0:(fld_handler.c:275:fld_server_lookup()) srv-fs9532-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [10648.983674] LustreError: 285193:0:(fld_handler.c:275:fld_server_lookup()) Skipped 57365 previous similar messages [10649.009792] LustreError: 285193:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) fs9532-MDT0001: Trash dir setup failed: rc = -5 [10649.024623] LustreError: 285193:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 57365 previous similar messages [10655.048808] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10660.671780] Lustre: server umount fs9532-MDT0001 complete [10690.031225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10694.663447] Lustre: server umount lustre-OST0000 complete [10705.775556] Lustre: server umount lustre-MDT0000 complete [10709.483431] LustreError: 280455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10709.508130] LustreError: 280455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 118 previous similar messages [10710.451420] LustreError: 277308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337527 with bad export cookie 13268599411793911680 [10710.453424] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10710.480256] LustreError: 277308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10711.007699] Lustre: server umount lustre-MDT0001 complete [10723.209419] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [10726.145542] Key type lgssc unregistered [10726.576201] LNet: 287503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10726.580433] LNetError: 287503:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10727.657393] LNet: Removed LNI 192.168.202.120@tcp [10728.661168] Key type .llcrypt unregistered [10728.663546] Key type ._llcrypt unregistered [10748.954239] Key type ._llcrypt registered [10748.956948] Key type .llcrypt registered [10749.060991] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [10750.273716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10750.444626] alg: No test for adler32 (adler32-zlib) [10751.564762] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [10751.869288] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [10753.567182] Key type lgssc registered [10754.915333] Lustre: Echo OBD driver; http://www.lustre.org/ [10763.311211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10763.333645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10764.949776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10768.658072] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10774.506538] LustreError: 288757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10774.520830] LustreError: 288757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10777.040381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10777.474074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10781.475672] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10786.274874] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10790.122949] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10797.253103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10797.620572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10802.721565] LustreError: 290365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10803.044657] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10807.786206] LustreError: 290364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10808.943814] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10809.831864] LustreError: 290365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10809.853058] LustreError: 290365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10814.982948] LustreError: 290672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10815.014404] LustreError: 290672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10823.155910] LustreError: 290672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10823.188782] LustreError: 290672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10830.295230] loop1: detected capacity change from 0 to 204800000 [10830.516659] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10830.539261] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10830.564584] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10837.476602] loop1: detected capacity change from 0 to 204800000 [10842.638869] LustreError: 290922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10842.676501] LustreError: 290922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [10844.177519] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10844.378649] Lustre: fs29035-OST0000: new disk, initializing [10844.385091] Lustre: srv-fs29035-OST0000: No data found on store. Initialize space. [10844.395350] Lustre: 292204:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10844.508847] Lustre: fs29035-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10851.400951] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10860.941442] Lustre: server umount fs29035-OST0000 complete [10875.681607] loop1: detected capacity change from 0 to 204800000 [10877.411353] LustreError: 290672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10877.425500] LustreError: 290672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [10881.996478] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10882.225710] Lustre: fs29035-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10882.537896] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10887.651804] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10888.176982] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10898.136083] loop2: detected capacity change from 0 to 204800000 [10898.471091] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10898.495078] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10898.519573] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10904.341785] loop2: detected capacity change from 0 to 204800000 [10910.668782] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10910.818874] LustreError: fs29035-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10910.830802] LustreError: fs29035-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10910.839831] LustreError: 295210:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -13 [10910.846070] LustreError: 295210:0:(tgt_mount.c:2084:server_put_super()) no obd fs29035-OST0001 [10910.852399] LustreError: 295210:0:(tgt_mount.c:116:server_deregister_mount()) fs29035-OST0001 not registered [10910.952913] Lustre: server umount fs29035-OST0001 complete [10910.956925] LustreError: 295210:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10922.145757] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10922.241611] Lustre: fs29035-OST0001: new disk, initializing [10922.246066] Lustre: srv-fs29035-OST0001: No data found on store. Initialize space. [10922.252622] Lustre: 295761:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10922.315092] Lustre: fs29035-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10928.082554] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10937.559883] Lustre: server umount fs29035-OST0000 complete [10943.982533] LustreError: 290364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10944.022796] LustreError: 290364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [10947.975850] Lustre: server umount fs29035-OST0001 complete [10960.229063] loop1: detected capacity change from 0 to 204800000 [10965.989495] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10966.283593] Lustre: fs29035-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10973.042994] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10980.633181] loop2: detected capacity change from 0 to 204800000 [10986.753719] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10994.966045] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11005.948606] Lustre: server umount fs29035-OST0000 complete [11016.350425] Lustre: server umount fs29035-OST0001 complete [11029.984693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11029.993235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11030.002299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11031.019964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11031.033708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11035.667855] Lustre: server umount lustre-OST0000 complete [11041.251077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11041.282492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11041.284882] Lustre: Skipped 1 previous similar message [11046.613475] Lustre: server umount lustre-MDT0000 complete [11050.685258] LustreError: 288738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337867 with bad export cookie 7488335041836628022 [11050.689424] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11050.700436] LustreError: 288738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11051.280917] Lustre: server umount lustre-MDT0001 complete [11064.177443] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [11067.254070] Key type lgssc unregistered [11067.665095] LNet: 301618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11067.675441] LNetError: 301618:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11067.701467] LNet: Removed LNI 192.168.202.120@tcp [11068.936538] Key type .llcrypt unregistered [11068.941066] Key type ._llcrypt unregistered [11110.443136] Key type ._llcrypt registered [11110.449919] Key type .llcrypt registered [11110.579495] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [11126.942324] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [11128.138429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11128.146905] alg: No test for adler32 (adler32-zlib) [11129.288717] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [11129.480526] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [11131.159185] Key type lgssc registered [11132.112940] Lustre: Echo OBD driver; http://www.lustre.org/ [11173.694973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11173.733428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11175.073968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11175.120485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11175.194378] Lustre: lustre-MDT0000: new disk, initializing [11175.257387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11175.275079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11180.215195] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11192.290075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11192.424886] Lustre: 305449:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11192.475158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11192.478617] Lustre: Skipped 1 previous similar message [11192.572915] Lustre: lustre-MDT0001: new disk, initializing [11192.673300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11192.703068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11192.710413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11197.504867] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11204.859199] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11210.518584] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11218.759424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11219.092414] Lustre: lustre-OST0000: new disk, initializing [11219.100122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11219.115691] Lustre: 307192:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11219.269851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11226.723979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11226.733763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11226.882675] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11227.234566] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11236.751825] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11240.633688] Lustre: Mounted lustre-client [11242.025421] LustreError: 307943:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8976465bd800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [11242.111332] Lustre: Unmounted lustre-client [11255.777830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11255.833804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11255.851481] Lustre: Skipped 1 previous similar message [11260.914056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11260.935925] Lustre: Skipped 1 previous similar message [11266.029109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11266.040513] Lustre: Skipped 1 previous similar message [11267.042248] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11267.483308] Lustre: server umount lustre-OST0000 complete [11274.724942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11274.742535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11274.767117] Lustre: Skipped 1 previous similar message [11274.776095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11276.258695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11279.502099] Lustre: server umount lustre-MDT0000 complete [11283.679045] LustreError: 305441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338100 with bad export cookie 10236307990210912040 [11283.690941] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11284.060654] Lustre: server umount lustre-MDT0001 complete [11297.190732] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [11299.487219] LNet: 1 local NIs in recovery (showing 1): 192.168.202.120@tcp [11301.221865] Key type lgssc unregistered [11301.551454] LNet: 309224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11301.568180] LNetError: 309224:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11372.262512] LNet: Removed LNI 192.168.202.120@tcp [11373.387584] Key type .llcrypt unregistered [11373.389112] Key type ._llcrypt unregistered [11396.173320] Key type ._llcrypt registered [11396.176351] Key type .llcrypt registered [11396.312722] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [11414.193801] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [11415.790041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11416.010616] alg: No test for adler32 (adler32-zlib) [11417.158152] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [11417.397261] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [11419.119345] Key type lgssc registered [11420.373469] Lustre: Echo OBD driver; http://www.lustre.org/ [11456.536481] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 14:24:31 (1780338271) [11464.655262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11464.676510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11466.030702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11466.079873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11466.239749] Lustre: lustre-MDT0000: new disk, initializing [11466.524805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11466.558464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11470.936342] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11484.664588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11484.903280] Lustre: 312429:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11484.981362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11484.988575] Lustre: Skipped 1 previous similar message [11485.180479] Lustre: lustre-MDT0001: new disk, initializing [11485.316559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11485.367206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11485.388039] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11492.005644] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11501.440491] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11504.620482] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11512.519284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11513.000667] Lustre: lustre-OST0000: new disk, initializing [11513.014843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11513.025445] Lustre: 314174:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11513.130144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11520.753691] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11521.144065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11521.151287] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11521.277882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11528.189859] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11541.986947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11541.991181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11542.020783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11542.037943] Lustre: Skipped 1 previous similar message [11544.878975] Lustre: server umount lustre-OST0000 complete [11551.714679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11551.727403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11551.743176] Lustre: Skipped 1 previous similar message [11551.766433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11555.380954] Lustre: server umount lustre-MDT0000 complete [11559.640668] LustreError: 312421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338376 with bad export cookie 6107235432624255661 [11559.665284] LustreError: 312421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11559.670192] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11559.932584] Lustre: server umount lustre-MDT0001 complete [11573.435606] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [11576.157816] Key type lgssc unregistered [11576.456634] LNet: 315806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11576.469773] LNetError: 315806:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11576.499791] LNet: Removed LNI 192.168.202.120@tcp [11577.455823] Key type .llcrypt unregistered [11577.458869] Key type ._llcrypt unregistered [11606.996550] Key type ._llcrypt registered [11606.999220] Key type .llcrypt registered [11607.109832] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [11608.550990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11608.682862] alg: No test for adler32 (adler32-zlib) [11609.753943] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [11609.953902] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [11611.615151] Key type lgssc registered [11612.733537] Lustre: Echo OBD driver; http://www.lustre.org/ [11621.396169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11621.415040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11622.931701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11626.789800] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11633.139603] LustreError: 317213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.170130] LustreError: 317213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11635.054628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11635.404214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11639.895826] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11645.283552] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11648.775829] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11655.370886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11655.734895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11662.421431] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11669.304784] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11678.069913] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11679.982731] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11686.883738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11686.910229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11686.932427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11687.480109] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.20@tcp (stopping) [11688.423521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11690.672003] Lustre: server umount lustre-MDT0000 complete [11692.550195] LustreError: 317209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11694.838234] LustreError: 317193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338511 with bad export cookie 9238464246249102690 [11694.857215] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11694.866896] LustreError: 317193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11697.668386] LustreError: 319268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11697.693561] LustreError: 319268:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11697.716609] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.20@tcp (stopping) [11697.720819] Lustre: Skipped 3 previous similar messages [11698.658800] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11698.681258] Lustre: Skipped 1 previous similar message [11699.682552] LustreError: 317210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11699.700810] LustreError: 317210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11701.356410] Lustre: server umount lustre-MDT0001 complete [11706.266089] Lustre: server umount lustre-OST0000 complete [11716.930348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11722.844899] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11725.816454] Lustre: 320940:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.120@tcp [11727.742860] Lustre: server umount MGS complete [11737.243580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11737.625491] LustreError: 321508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11737.862803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11743.420889] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11747.854279] LustreError: 321510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11747.881908] LustreError: 321510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11753.661263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11754.103644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11758.216533] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11767.323500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11767.677166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11774.842822] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11780.788351] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11782.541536] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11786.618732] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11788.271526] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11793.382112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11793.401739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11793.406873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11793.406882] Lustre: Skipped 2 previous similar messages [11798.496539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11798.504787] Lustre: Skipped 1 previous similar message [11799.272152] Lustre: server umount lustre-OST0000 complete [11803.632088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11803.652305] Lustre: Skipped 1 previous similar message [11808.736989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11808.746641] Lustre: Skipped 4 previous similar messages [11809.638439] Lustre: server umount lustre-MDT0000 complete [11813.831571] LustreError: 321494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338630 with bad export cookie 9238464246249103523 [11813.848554] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11813.850630] LustreError: 321494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11814.185849] Lustre: server umount lustre-MDT0001 complete [11824.981942] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [11828.074621] Key type lgssc unregistered [11828.350902] LNet: 324690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11828.361751] LNetError: 324690:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11828.383112] LNet: Removed LNI 192.168.202.120@tcp [11829.161845] Key type .llcrypt unregistered [11829.163497] Key type ._llcrypt unregistered [11851.939593] Key type ._llcrypt registered [11851.941733] Key type .llcrypt registered [11852.060868] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [11869.733747] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [11870.766255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11870.910958] alg: No test for adler32 (adler32-zlib) [11872.090408] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [11872.342777] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [11874.088140] Key type lgssc registered [11875.487242] Lustre: Echo OBD driver; http://www.lustre.org/ [11914.996404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11915.012181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11916.490080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11916.582756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11916.641386] Lustre: lustre-MDT0000: new disk, initializing [11916.710599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11916.742078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11921.542826] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11932.496554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11932.629641] Lustre: 327823:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11932.697608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11932.704130] Lustre: Skipped 1 previous similar message [11932.804207] Lustre: lustre-MDT0001: new disk, initializing [11932.907117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11932.967645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11932.983878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11939.709788] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11947.877897] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11951.666394] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11957.924377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11958.212957] Lustre: lustre-OST0000: new disk, initializing [11958.218084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11958.229170] Lustre: 329568:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11958.358647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11964.805656] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11966.101250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11966.116721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11966.250385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11972.330577] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11976.915639] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11977.091945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11981.693946] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11981.923314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11986.937727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11986.940538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11986.958148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11986.959162] LustreError: Skipped 1 previous similar message [11986.988425] Lustre: Skipped 1 previous similar message [11990.494527] Lustre: server umount lustre-OST0000 complete [11994.593046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11994.605179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11994.621883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11994.631055] Lustre: Skipped 1 previous similar message [11997.152926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11997.158234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12000.692444] Lustre: server umount lustre-MDT0000 complete [12004.621976] LustreError: 327815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338821 with bad export cookie 4710717304627332234 [12004.628282] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12004.631887] LustreError: 327815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12004.884715] Lustre: server umount lustre-MDT0001 complete [12013.582492] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 14:33:48 (1780338828) [12029.245934] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [12039.706720] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [12080.590247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12080.864589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12080.918807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12081.039730] Lustre: lustre-MDT0000: new disk, initializing [12081.178692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12081.213350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12086.018839] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12097.657416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12097.788539] Lustre: 334142:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12097.873967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12097.876468] Lustre: Skipped 1 previous similar message [12097.947682] Lustre: lustre-MDT0001: new disk, initializing [12098.052188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12098.108204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12098.124921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12102.969145] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12111.304607] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12115.678619] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12122.957355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12123.247532] Lustre: lustre-OST0000: new disk, initializing [12123.250725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12123.257426] Lustre: 335887:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12123.361602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12128.858261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12128.868714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12129.029686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12130.915982] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12140.479246] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12185.574193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12185.574579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12185.590325] LustreError: Skipped 1 previous similar message [12185.596544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12185.596553] Lustre: Skipped 1 previous similar message [12185.621736] Lustre: Skipped 1 previous similar message [12186.701780] Lustre: server umount lustre-OST0000 complete [12195.297390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12195.304332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12195.319513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12195.323229] Lustre: Skipped 1 previous similar message [12197.706106] Lustre: server umount lustre-MDT0000 complete [12200.928537] LustreError: 334940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12200.958699] LustreError: 334940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12203.248740] LustreError: 334136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339019 with bad export cookie 4710717304627333158 [12203.253407] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12203.264909] LustreError: 334136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12204.230992] Lustre: server umount lustre-MDT0001 complete [12219.308224] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12222.301916] Key type lgssc unregistered [12222.610631] LNet: 337933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12222.615953] LNetError: 337933:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12222.632346] LNet: Removed LNI 192.168.202.120@tcp [12223.551191] Key type .llcrypt unregistered [12223.553067] Key type ._llcrypt unregistered [12331.035779] Key type ._llcrypt registered [12331.043448] Key type .llcrypt registered [12331.183244] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12331.501619] Key type .llcrypt unregistered [12331.504840] Key type ._llcrypt unregistered [12423.808733] Key type ._llcrypt registered [12423.815842] Key type .llcrypt registered [12423.972621] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12424.328957] Key type .llcrypt unregistered [12424.336301] Key type ._llcrypt unregistered [12463.972300] Key type ._llcrypt registered [12463.976981] Key type .llcrypt registered [12464.238164] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12464.730281] Key type .llcrypt unregistered [12464.733558] Key type ._llcrypt unregistered [12501.863398] Key type ._llcrypt registered [12501.870275] Key type .llcrypt registered [12501.994966] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12502.328458] Key type .llcrypt unregistered [12502.332995] Key type ._llcrypt unregistered [12587.566090] Key type ._llcrypt registered [12587.568804] Key type .llcrypt registered [12587.705464] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12588.071608] Key type .llcrypt unregistered [12588.073761] Key type ._llcrypt unregistered [12624.968869] Key type ._llcrypt registered [12624.981498] Key type .llcrypt registered [12625.094845] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12625.475540] Key type .llcrypt unregistered [12625.478256] Key type ._llcrypt unregistered [12662.035990] Key type ._llcrypt registered [12662.044124] Key type .llcrypt registered [12662.156663] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12662.469272] Key type .llcrypt unregistered [12662.470531] Key type ._llcrypt unregistered [12696.610513] Key type ._llcrypt registered [12696.614707] Key type .llcrypt registered [12696.810720] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12697.264374] Key type .llcrypt unregistered [12697.271194] Key type ._llcrypt unregistered [12738.219368] Key type ._llcrypt registered [12738.221574] Key type .llcrypt registered [12738.462905] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12738.876315] Key type .llcrypt unregistered [12738.878133] Key type ._llcrypt unregistered [12783.688569] Key type ._llcrypt registered [12783.690393] Key type .llcrypt registered [12783.766475] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12784.029474] Key type .llcrypt unregistered [12784.033585] Key type ._llcrypt unregistered [12851.969084] Key type ._llcrypt registered [12851.971595] Key type .llcrypt registered [12852.063315] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12852.330324] Key type .llcrypt unregistered [12852.331927] Key type ._llcrypt unregistered [12889.749331] Key type ._llcrypt registered [12889.751883] Key type .llcrypt registered [12889.881268] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12890.290720] Key type .llcrypt unregistered [12890.293083] Key type ._llcrypt unregistered [12921.299833] Key type ._llcrypt registered [12921.302682] Key type .llcrypt registered [12921.425413] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12921.701549] Key type .llcrypt unregistered [12921.709123] Key type ._llcrypt unregistered [12981.614880] Key type ._llcrypt registered [12981.617335] Key type .llcrypt registered [12981.735405] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [13001.677101] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [13003.227389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13003.262093] alg: No test for adler32 (adler32-zlib) [13004.503085] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [13004.828220] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [13006.639200] Key type lgssc registered [13007.742322] Lustre: Echo OBD driver; http://www.lustre.org/ [13056.361069] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13056.423467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13057.849088] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13057.900985] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13058.054451] Lustre: tst250fs-MDT0000: new disk, initializing [13058.161574] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13058.193564] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13064.279741] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13078.376563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13078.525599] Lustre: 358046:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13078.615500] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13078.622446] Lustre: Skipped 1 previous similar message [13078.718236] Lustre: tst250fs-MDT0001: new disk, initializing [13078.824890] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13078.945575] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13078.966738] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13084.198292] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13089.497104] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13097.339764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13097.809776] Lustre: tst250fs-OST0000: new disk, initializing [13097.820665] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13097.841955] Lustre: 359648:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13098.012618] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13104.494266] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13105.265346] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13105.276159] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13105.393399] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13118.123706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13118.376432] Lustre: tst250fs-OST0001: new disk, initializing [13118.381415] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13118.397333] Lustre: 360502:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13118.522339] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13127.075933] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13128.782991] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13128.806742] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13128.939696] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13133.243332] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13135.308864] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13135.315421] Lustre: Skipped 1 previous similar message [13155.809805] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13155.815458] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13155.829513] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13159.915424] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13159.939926] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13159.945548] Lustre: Skipped 1 previous similar message [13159.955817] Lustre: Skipped 2 previous similar messages [13161.677804] Lustre: server umount tst250fs-MDT0000 complete [13165.515094] LustreError: 358040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339982 with bad export cookie 49211854527187056 [13165.534611] LustreError: 358040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13165.542162] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13166.085587] Lustre: server umount tst250fs-MDT0001 complete [13180.971735] Lustre: server umount tst250fs-OST0000 complete [13195.450927] Lustre: server umount tst250fs-OST0001 complete [13215.849466] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [13226.450867] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [13266.112407] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 14:54:40 (1780340080) [13268.113430] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13288.518140] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [13300.964394] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [13337.693882] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13075 sec ========= 14:55:52 (1780340152) [13339.559529] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:55:54 (1780340154) === [13343.620406] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:55:58 (1780340158) === [13391.704282] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [13394.850302] Key type lgssc unregistered [13395.117821] LNet: 369536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13395.122596] LNetError: 369536:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13395.140681] LNet: Removed LNI 192.168.202.120@tcp [13396.076684] Key type .llcrypt unregistered [13396.078428] Key type ._llcrypt unregistered