[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 519339373 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002370] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004021] kvm-guest: setup PV IPIs [ 0.007227] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010107] LSM: Security Framework initializing [ 0.011043] Yama: becoming mindful. [ 0.012023] SELinux: Initializing. [ 0.013045] *** VALIDATE selinux *** [ 0.021343] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025883] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029100] *** VALIDATE tmpfs *** [ 0.030359] *** VALIDATE proc *** [ 0.031187] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.034242] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038014] Speculative Store Bypass: Vulnerable [ 0.041430] debug: unmapping init [mem 0xffffffff89a59000-0xffffffff89a60fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044667] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045026] ... version: 2 [ 0.046017] ... bit width: 48 [ 0.047016] ... generic registers: 4 [ 0.048017] ... value mask: 0000ffffffffffff [ 0.049018] ... max period: 00007fffffffffff [ 0.050017] ... fixed-purpose events: 3 [ 0.051015] ... event mask: 000000070000000f [ 0.052340] rcu: Hierarchical SRCU implementation. [ 0.054446] smp: Bringing up secondary CPUs ... [ 0.055643] x86: Booting SMP configuration: [ 0.056031] .... node #0, CPUs: #1 #2 #3 [ 0.059597] smp: Brought up 1 node, 4 CPUs [ 0.061017] smpboot: Max logical packages: 1 [ 0.062025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.240031] node 0 deferred pages initialised in 175ms [ 0.245050] devtmpfs: initialized [ 0.246340] x86/mm: Memory block size: 128MB [ 0.250584] gcov: version magic: 0x41383552 [ 0.254151] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.255088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.257357] pinctrl core: initialized pinctrl subsystem [ 0.259291] [ 0.259895] ************************************************************* [ 0.263019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.265012] ** ** [ 0.267013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.269017] ** ** [ 0.272018] ** This means that this kernel is built to expose internal ** [ 0.275022] ** IOMMU data structures, which may compromise security on ** [ 0.277018] ** your system. ** [ 0.279020] ** ** [ 0.282018] ** If you see this message and you are not debugging the ** [ 0.284013] ** kernel, report this immediately to your vendor! ** [ 0.287032] ** ** [ 0.289017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.292017] ************************************************************* [ 0.295345] NET: Registered protocol family 16 [ 0.297525] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.300083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.303089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.308227] cpuidle: using governor menu [ 0.310696] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.313581] PCI: Using configuration type 1 for base access [ 0.315163] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.325132] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.327053] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.331077] cryptd: max_cpu_qlen set to 1000 [ 0.333295] ACPI: Added _OSI(Module Device) [ 0.335019] ACPI: Added _OSI(Processor Device) [ 0.337018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.339019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.343708] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.351330] ACPI: Interpreter enabled [ 0.353076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.354018] ACPI: Using IOAPIC for interrupt routing [ 0.356167] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.360499] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.371203] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.373055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.376031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.379138] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.384679] acpiphp: Slot [2] registered [ 0.386178] acpiphp: Slot [5] registered [ 0.388218] acpiphp: Slot [6] registered [ 0.390183] acpiphp: Slot [7] registered [ 0.392167] acpiphp: Slot [8] registered [ 0.393164] acpiphp: Slot [9] registered [ 0.395177] acpiphp: Slot [10] registered [ 0.396172] acpiphp: Slot [3] registered [ 0.398125] acpiphp: Slot [4] registered [ 0.399145] acpiphp: Slot [11] registered [ 0.401111] acpiphp: Slot [12] registered [ 0.402140] acpiphp: Slot [13] registered [ 0.404130] acpiphp: Slot [14] registered [ 0.405123] acpiphp: Slot [15] registered [ 0.407112] acpiphp: Slot [16] registered [ 0.408144] acpiphp: Slot [17] registered [ 0.410113] acpiphp: Slot [18] registered [ 0.412143] acpiphp: Slot [19] registered [ 0.413155] acpiphp: Slot [20] registered [ 0.415132] acpiphp: Slot [21] registered [ 0.416121] acpiphp: Slot [22] registered [ 0.418128] acpiphp: Slot [23] registered [ 0.419128] acpiphp: Slot [24] registered [ 0.421178] acpiphp: Slot [25] registered [ 0.423107] acpiphp: Slot [26] registered [ 0.424096] acpiphp: Slot [27] registered [ 0.425112] acpiphp: Slot [28] registered [ 0.427130] acpiphp: Slot [29] registered [ 0.428133] acpiphp: Slot [30] registered [ 0.430140] acpiphp: Slot [31] registered [ 0.431077] PCI host bridge to bus 0000:00 [ 0.433025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.435031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.437030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.440054] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.442043] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.445054] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.447215] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.450013] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.454459] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.463776] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.468519] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.471019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.473016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.475027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.478172] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.480869] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.484069] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.486897] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.491020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.503024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.506996] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.513000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.519018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.528017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.545020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.555454] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.563023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.569028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.589037] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.601084] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.608026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.616025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.634020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.645308] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.656022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.664022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.682031] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.699732] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.710016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.715019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.731019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.744570] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.756028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.764022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.776028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.786702] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.788401] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.791409] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.793407] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.796320] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.800563] iommu: Default domain type: Passthrough [ 0.802593] SCSI subsystem initialized [ 0.804207] ACPI: bus type USB registered [ 0.805242] usbcore: registered new interface driver usbfs [ 0.809123] usbcore: registered new interface driver hub [ 0.810076] usbcore: registered new device driver usb [ 0.812212] pps_core: LinuxPPS API ver. 1 registered [ 0.814012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.816063] PTP clock support registered [ 0.819055] EDAC MC: Ver: 3.0.0 [ 0.820634] PCI: Using ACPI for IRQ routing [ 0.823634] NetLabel: Initializing [ 0.824024] NetLabel: domain hash size = 128 [ 0.824923] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.826116] NetLabel: unlabeled traffic allowed by default [ 0.828230] vgaarb: loaded [ 0.830332] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.832022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.839206] clocksource: Switched to clocksource kvm-clock [ 0.942001] VFS: Disk quotas dquot_6.6.0 [ 0.943736] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.946596] *** VALIDATE ramfs *** [ 0.947966] *** VALIDATE hugetlbfs *** [ 0.949580] pnp: PnP ACPI init [ 0.952058] pnp: PnP ACPI: found 6 devices [ 0.969187] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.972544] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.974240] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.976376] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.977935] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.979864] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.982083] NET: Registered protocol family 2 [ 0.984722] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.989594] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.993097] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.998125] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.001538] TCP: Hash tables configured (established 65536 bind 65536) [ 1.004271] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.007215] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.009928] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.013097] NET: Registered protocol family 1 [ 1.015690] RPC: Registered named UNIX socket transport module. [ 1.017684] RPC: Registered udp transport module. [ 1.019412] RPC: Registered tcp transport module. [ 1.021089] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.023559] NET: Registered protocol family 44 [ 1.025378] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.027899] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.030379] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.032605] PCI: CLS 0 bytes, default 64 [ 1.034527] Unpacking initramfs... [ 2.458277] debug: unmapping init [mem 0xffff8df5fcc54000-0xffff8df5fffbffff] [ 2.462192] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.464029] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.466601] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.985789] Initialise system trusted keyrings [ 2.987398] Key type blacklist registered [ 2.989566] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.999665] zbud: loaded [ 3.002734] *** VALIDATE nfs *** [ 3.004040] *** VALIDATE nfs4 *** [ 3.005413] pstore: using deflate compression [ 3.009098] Platform Keyring initialized [ 3.123130] NET: Registered protocol family 38 [ 3.124885] Key type asymmetric registered [ 3.126730] Asymmetric key parser 'x509' registered [ 3.128885] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.132200] io scheduler mq-deadline registered [ 3.133974] io scheduler kyber registered [ 3.135504] io scheduler bfq registered [ 3.137383] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.140319] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.142961] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.145716] ACPI: Power Button [PWRF] [ 3.151300] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.158456] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.174674] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.182164] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.198654] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.227767] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.256602] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.260828] Non-volatile memory driver v1.3 [ 3.261953] Linux agpgart interface v0.103 [ 3.299513] virtio_blk virtio1: [vda] 155184 512-byte logical blocks (79.5 MB/75.8 MiB) [ 3.303150] vda: detected capacity change from 0 to 79454208 [ 3.321254] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.324563] vdb: detected capacity change from 0 to 1073741824 [ 3.343481] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.346725] vdc: detected capacity change from 0 to 2621440000 [ 3.366237] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.369382] vdd: detected capacity change from 0 to 2621440000 [ 3.383681] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.385628] vde: detected capacity change from 0 to 4294967296 [ 3.397136] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.399162] vdf: detected capacity change from 0 to 4294967296 [ 3.404304] libphy: Fixed MDIO Bus: probed [ 3.408694] usbcore: registered new interface driver usbserial_generic [ 3.410524] usbserial: USB Serial support registered for generic [ 3.412141] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.415180] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.416415] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.418507] mousedev: PS/2 mouse device common for all mice [ 3.420538] rtc_cmos 00:05: RTC can wake from S4 [ 3.422733] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.426119] rtc_cmos 00:05: registered as rtc0 [ 3.427709] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.429637] intel_pstate: CPU model not supported [ 3.431898] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.436050] hid: raw HID events driver (C) Jiri Kosina [ 3.436597] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.438498] usbcore: registered new interface driver usbhid [ 3.443423] usbhid: USB HID core driver [ 3.445390] drop_monitor: Initializing network drop monitor service [ 3.448859] Initializing XFRM netlink socket [ 3.451093] NET: Registered protocol family 10 [ 3.454039] Segment Routing with IPv6 [ 3.455085] NET: Registered protocol family 17 [ 3.457174] mpls_gso: MPLS GSO support [ 3.462228] RAS: Correctable Errors collector initialized. [ 3.464583] AVX version of gcm_enc/dec engaged. [ 3.467260] AES CTR mode by8 optimization enabled [ 3.549601] sched_clock: Marking stable (3549574283, 0)->(4533679450, -984105167) [ 3.553717] registered taskstats version 1 [ 3.556242] Loading compiled-in X.509 certificates [ 3.558566] zswap: loaded using pool lzo/zbud [ 3.583145] Key type big_key registered [ 3.596980] Key type encrypted registered [ 3.598450] ima: No TPM chip found, activating TPM-bypass! [ 3.600077] ima: Allocated hash algorithm: sha1 [ 3.601499] ima: No architecture policies found [ 3.603118] evm: Initialising EVM extended attributes: [ 3.604622] evm: security.selinux [ 3.605589] evm: security.ima [ 3.606303] evm: security.capability [ 3.607343] evm: HMAC attrs: 0x1 [ 3.609164] rtc_cmos 00:05: setting system clock to 2026-09-09 03:57:07 UTC (1788926227) [ 3.614588] debug: unmapping init [mem 0xffffffff8aa03000-0xffffffff8abfffff] [ 3.617588] debug: unmapping init [mem 0xffffffff89782000-0xffffffff89a58fff] [ 3.627084] Write protecting the kernel read-only data: 28672k [ 3.630146] debug: unmapping init [mem 0xffffffff87e03000-0xffffffff87ffffff] [ 3.632437] debug: unmapping init [mem 0xffffffff88714000-0xffffffff887fffff] [ 3.667871] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.674738] systemd[1]: Detected virtualization kvm. [ 3.676660] systemd[1]: Detected architecture x86-64. [ 3.678242] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.714708] systemd[1]: No hostname configured. [ 3.715910] systemd[1]: Set hostname to . [ 3.724029] random: systemd: uninitialized urandom read (16 bytes read) [ 3.726360] systemd[1]: Initializing machine ID from random generator. [ 3.791482] random: ln: uninitialized urandom read (6 bytes read) [ 3.881027] random: systemd: uninitialized urandom read (16 bytes read) [ 3.884166] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.888773] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.893553] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.577348] device-mapper: uevent: version 1.0.3 [ 4.579133] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.325967] random: fast init done [ 5.348642] virtio_net virtio0 ens2: renamed from eth0 [ 5.405869] scsi host0: ata_piix [ 5.438243] scsi host1: ata_piix [ 5.439598] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.441839] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.080727] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.112290] random: crng init done [ 10.113656] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.584066] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.743212] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.054726] SELinux: Disabled at runtime. [ 12.116286] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.124214] systemd[1]: Detected virtualization kvm. [ 12.125986] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.642203] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.646570] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.650720] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.654065] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.656100] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.661758] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.669233] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ 12.748425] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.035978] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.294934] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.335433] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.440149] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.461698] EDAC sbridge: Ver: 1.1.2 [ 14.800348] Key type dns_resolver registered [ 15.085154] NFS: Registering the id_resolver key type [ 15.087361] Key type id_resolver registered [ 15.089010] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg249-server login: [ 41.584074] libcfs: loading out-of-tree module taints kernel. [ 41.608453] Key type ._llcrypt registered [ 41.610021] Key type .llcrypt registered [ 41.669303] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 50.909736] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 51.691681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.700785] alg: No test for adler32 (adler32-zlib) [ 52.748764] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 53.116409] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 54.760257] Key type lgssc registered [ 56.100752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.034307] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 130.006712] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 145.216474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.254293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 145.440124] hrtimer: interrupt took 4114490 ns [ 146.605835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 146.644083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 146.755936] Lustre: lustre-MDT0000: new disk, initializing [ 146.874395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 146.903151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 153.389736] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.940504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.090297] Lustre: 6487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.124262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.130423] Lustre: Skipped 1 previous similar message [ 169.240318] Lustre: lustre-MDT0001: new disk, initializing [ 169.302250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 169.339193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 169.351586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.843820] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.890915] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 189.784907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.046951] Lustre: lustre-OST0000: new disk, initializing [ 190.051638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.058795] Lustre: 8427:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.149288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.561226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.570380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.621186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 196.489624] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.760350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.853701] Lustre: lustre-OST0001: new disk, initializing [ 211.860613] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 211.867178] Lustre: 9500:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 211.927050] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.044793] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.223898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.253526] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 219.360889] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 235.243976] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 243.020290] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 250.029577] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing check_logdir /tmp/testlogs/ [ 255.028786] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing yml_node [ 259.950512] Lustre: DEBUG MARKER: Client: 2.17.57.55 [ 262.511635] Lustre: DEBUG MARKER: MDS: 2.17.57.55 [ 265.665230] Lustre: DEBUG MARKER: OSS: 2.17.57.55 [ 267.418777] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 9 00:01:29 EDT 2026 [ 287.084142] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 289.014768] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 306.657023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.663070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.671468] Lustre: Skipped 2 previous similar messages [ 306.691756] Lustre: Skipped 3 previous similar messages [ 311.907508] Lustre: server umount lustre-MDT0000 complete [ 315.732453] LustreError: 6479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788926539 with bad export cookie 1227606131744519820 [ 315.736072] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.742337] LustreError: 6479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 316.898254] LustreError: 6492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.898587] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.898819] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.922413] LustreError: 6492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 316.954444] Lustre: Skipped 2 previous similar messages [ 320.930567] LustreError: 6492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.932461] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.946369] LustreError: 6492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 320.966404] Lustre: Skipped 2 previous similar messages [ 326.068525] LustreError: 6494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.069058] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.097432] LustreError: 6494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 326.126521] Lustre: Skipped 1 previous similar message [ 330.208189] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 330.542666] Lustre: server umount lustre-MDT0001 complete [ 334.806518] Lustre: server umount lustre-OST0000 complete [ 339.975505] Lustre: server umount lustre-OST0001 complete [ 348.809643] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 359.026273] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 403.933485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.482427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 404.562873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 404.681093] Lustre: lustre-MDT0000: new disk, initializing [ 404.745657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 404.765573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 410.856979] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 422.325211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.445351] Lustre: 15068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 422.493957] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 422.498440] Lustre: Skipped 1 previous similar message [ 422.572621] Lustre: lustre-MDT0001: new disk, initializing [ 422.700016] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 422.752236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 422.757511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 428.886326] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 441.713737] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 448.340370] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 456.530365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 456.848788] Lustre: lustre-OST0000: new disk, initializing [ 456.852540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 456.859537] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 456.952442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 458.375262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 458.404363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 458.540649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 465.086625] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 476.656873] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 483.129633] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 483.498930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 488.758505] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 489.032067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 494.579565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 494.583923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.586190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 494.589734] LustreError: Skipped 1 previous similar message [ 494.625621] Lustre: Skipped 1 previous similar message [ 497.341307] Lustre: server umount lustre-OST0000 complete [ 504.802789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 504.803549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.840346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 504.851451] Lustre: Skipped 2 previous similar messages [ 507.976351] Lustre: server umount lustre-MDT0000 complete [ 509.922751] LustreError: 16934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.944839] LustreError: 16934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 511.805080] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788926735 with bad export cookie 1227606131744521353 [ 511.822411] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 512.096364] Lustre: server umount lustre-MDT0001 complete [ 518.889489] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 00:05:40 (1788926740) [ 541.083267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.470762] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.608284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.274950] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 551.915985] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.928727] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 558.312717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.044879] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 574.626200] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 581.518886] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 584.004878] Lustre: Failing over lustre-MDT0000 [ 584.370647] Lustre: server umount lustre-MDT0000 complete [ 584.679666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 584.694854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.705349] Lustre: Skipped 1 previous similar message [ 584.708594] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.727978] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 604.128250] Lustre: 3617:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788926812/real 1788926812] req@ffff8df546b6df80 x1875825160000896/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788926828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 604.181315] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 605.775597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.725512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.734308] Lustre: Skipped 1 previous similar message [ 614.791580] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 620.010139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 620.011295] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 620.066794] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 620.302872] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 631.267370] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 638.437601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.449904] Lustre: Skipped 1 previous similar message [ 638.457256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 638.469227] Lustre: Skipped 1 previous similar message [ 643.301821] Lustre: server umount lustre-MDT0000 complete [ 643.555450] LustreError: 19918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.571301] LustreError: 19918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 648.166137] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788926872 with bad export cookie 1227606131744522767 [ 648.183750] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 648.190675] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 648.532340] Lustre: server umount lustre-MDT0001 complete [ 659.779103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 666.537108] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 679.675685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.846482] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 697.461597] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 704.036714] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 707.140625] Lustre: Failing over lustre-MDT0000 [ 707.514459] Lustre: server umount lustre-MDT0000 complete [ 711.149886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 711.178200] LustreError: 23252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.179781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 711.194496] LustreError: 23252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 727.211664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.319125] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 727.571302] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 731.527248] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 732.655190] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 732.660993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 732.668496] Lustre: Skipped 1 previous similar message [ 732.688189] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 744.363625] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 753.125881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 753.145891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 753.162973] Lustre: Skipped 1 previous similar message [ 753.175792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 757.287844] Lustre: server umount lustre-MDT0000 complete [ 762.707657] LustreError: 23237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788926986 with bad export cookie 1227606131744523705 [ 762.715438] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 763.255588] Lustre: server umount lustre-MDT0001 complete [ 775.564906] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 00:09:56 (1788926996) [ 798.686899] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 812.978844] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 865.375564] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 879.362592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 879.695229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 879.748349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 879.849207] Lustre: lustre-MDT0000: new disk, initializing [ 879.947572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 879.959066] Lustre: Skipped 3 previous similar messages [ 879.989731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 886.265457] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 899.938294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 900.197840] Lustre: 29639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 900.528910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 906.766685] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 917.929750] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 924.760623] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 934.031734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 934.660287] Lustre: 31621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 936.225644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 936.256640] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 936.290941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 936.296949] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 943.477581] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 956.451183] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 964.120569] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 987.618380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 987.626946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.630359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 987.635097] Lustre: Skipped 2 previous similar messages [ 987.643030] Lustre: Skipped 2 previous similar messages [ 992.744961] LustreError: 31604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 992.763095] LustreError: 31604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 992.851897] Lustre: server umount lustre-OST0000 complete [ 1003.614219] Lustre: server umount lustre-MDT0000 complete [ 1008.067336] LustreError: 31620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788927231 with bad export cookie 1227606131744524111 [ 1008.068934] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1008.078195] LustreError: 31620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1008.665546] Lustre: server umount lustre-MDT0001 complete [ 1021.985086] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 1026.242766] Key type lgssc unregistered [ 1026.797558] LNet: 33604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.806733] LNetError: 33604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.831996] LNet: Removed LNI 192.168.202.149@tcp [ 1028.093553] Key type .llcrypt unregistered [ 1028.100201] Key type ._llcrypt unregistered [ 1055.719111] Key type ._llcrypt registered [ 1055.722070] Key type .llcrypt registered [ 1055.863299] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1057.350346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1057.413231] alg: No test for adler32 (adler32-zlib) [ 1058.838342] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 1059.188369] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 1060.864301] Key type lgssc registered [ 1062.326514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1072.159486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1072.182402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.882831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.441388] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1083.363116] LustreError: 35020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1083.384338] LustreError: 35020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1088.499739] LustreError: 35019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1092.182701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1092.881685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1098.548312] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1109.553526] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1117.216815] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1124.267482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1124.765937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.202805] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1138.131399] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1141.302879] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1144.357773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1165.698260] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:16:27 (1788927387) [ 1167.987883] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1170.532255] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1179.235085] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 00:16:41 (1788927401) [ 1189.990725] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 00:16:51 (1788927411) [ 1191.095603] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1201.888733] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1201.899246] Lustre: Skipped 2 previous similar messages [ 1204.359680] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 00:17:05 (1788927425) [ 1206.941239] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1206.946183] Lustre: Skipped 2 previous similar messages [ 1217.063785] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1217.070881] Lustre: Skipped 4 previous similar messages [ 1233.154375] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 1233.157827] Lustre: Skipped 5 previous similar messages [ 1235.806974] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 00:17:37 (1788927457) [ 1239.250231] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1243.548697] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1254.112671] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 00:17:55 (1788927475) [ 1270.481708] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1270.485964] Lustre: Skipped 4 previous similar messages [ 1272.966716] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 00:18:14 (1788927494) [ 1296.877773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1296.885072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.896230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1297.896376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1297.898189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1297.898573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.932154] Lustre: Skipped 1 previous similar message [ 1303.013528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1303.032314] Lustre: Skipped 1 previous similar message [ 1307.104480] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1307.295756] Lustre: server umount lustre-OST0000 complete [ 1313.250617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.251528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1313.273343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1313.289699] Lustre: Skipped 1 previous similar message [ 1318.370048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1318.373164] Lustre: Skipped 1 previous similar message [ 1318.665027] Lustre: server umount lustre-MDT0000 complete [ 1323.175844] LustreError: 39460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788927547 with bad export cookie 9550140620713202335 [ 1323.184995] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1323.500322] Lustre: server umount lustre-MDT0001 complete [ 1339.211437] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 1343.388680] Key type lgssc unregistered [ 1343.884379] LNet: 42900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1343.895339] LNetError: 42900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.932288] LNet: Removed LNI 192.168.202.149@tcp [ 1345.292344] Key type .llcrypt unregistered [ 1345.298380] Key type ._llcrypt unregistered [ 1368.276287] Key type ._llcrypt registered [ 1368.280787] Key type .llcrypt registered [ 1368.384706] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 1388.796710] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1389.853646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1389.910163] alg: No test for adler32 (adler32-zlib) [ 1391.419986] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 1391.839925] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 1393.648182] Key type lgssc registered [ 1395.406497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1443.408515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1443.462874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1444.825443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1444.875867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1444.988974] Lustre: lustre-MDT0000: new disk, initializing [ 1445.110477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1445.129836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1450.511577] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1462.506128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1462.775593] Lustre: 46056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1462.899318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1462.906964] Lustre: Skipped 1 previous similar message [ 1463.007373] Lustre: lustre-MDT0001: new disk, initializing [ 1463.235560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1463.305283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1463.328311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1468.018605] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1478.789749] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1486.050661] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1493.991030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1494.347251] Lustre: lustre-OST0000: new disk, initializing [ 1494.350487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1494.361873] Lustre: 48025:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1494.450393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1499.782329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1499.794974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1499.994236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1501.500317] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1511.575955] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1516.530666] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1516.877611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1523.002580] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1523.262286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1530.849095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1530.850301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1530.858860] LustreError: Skipped 1 previous similar message [ 1530.866421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1530.882598] Lustre: Skipped 1 previous similar message [ 1530.903803] Lustre: Skipped 1 previous similar message [ 1532.239120] Lustre: server umount lustre-OST0000 complete [ 1540.068499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1540.074488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1540.093832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1541.090619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.118377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1541.122328] Lustre: Skipped 1 previous similar message [ 1543.872963] Lustre: server umount lustre-MDT0000 complete [ 1548.357952] LustreError: 46047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788927772 with bad export cookie 17302987034850041644 [ 1548.374511] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1548.377424] LustreError: 46047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1549.128913] Lustre: server umount lustre-MDT0001 complete [ 1560.870697] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:23:02 (1788927782) [ 1569.687664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1570.065171] LustreError: 50490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1570.090037] LustreError: 50490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1570.219531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1575.401757] LustreError: 50491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1575.607654] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1579.494849] LustreError: 50490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1584.612688] LustreError: 50491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1585.694590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1586.167172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1590.794245] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1600.727143] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1608.840722] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1619.858561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1620.609671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.623774] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1638.095750] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1647.744394] LustreError: 53021:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1648.885532] Lustre: Modifying parameter general.timeout=1 in log params [ 1650.004267] Lustre: Modifying parameter general.timeout=2 in log params [ 1652.006988] Lustre: Modifying parameter general.timeout=4 in log params [ 1652.019726] Lustre: Skipped 1 previous similar message [ 1656.117141] Lustre: Modifying parameter general.timeout=8 in log params [ 1656.131511] Lustre: Skipped 3 previous similar messages [ 1665.026716] Lustre: Modifying parameter general.timeout=16 in log params [ 1665.033516] Lustre: Skipped 7 previous similar messages [ 1681.144433] Lustre: Modifying parameter general.timeout=29 in log params [ 1681.148448] Lustre: Skipped 12 previous similar messages [ 1713.679976] Lustre: Modifying parameter general.timeout=60 in log params [ 1713.684628] Lustre: Skipped 30 previous similar messages [ 1778.326763] Lustre: Modifying parameter general.timeout=116 in log params [ 1778.342850] Lustre: Skipped 55 previous similar messages [ 1805.288552] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:27:07 (1788928027) [ 1808.240203] Lustre: 59975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1810.430950] Lustre: 60088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1812.469778] Lustre: 60186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1817.144908] Lustre: 60383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1817.166167] Lustre: 60383:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1849.701464] Lustre: 61854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1849.714980] Lustre: 61854:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1849.757227] LustreError: 61854:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1856.914614] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:27:58 (1788928078) [ 1877.477245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.497417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.509395] Lustre: Skipped 1 previous similar message [ 1877.528110] Lustre: Skipped 2 previous similar messages [ 1882.593482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1882.609962] Lustre: Skipped 2 previous similar messages [ 1883.008211] Lustre: server umount lustre-MDT0000 complete [ 1887.719944] LustreError: 50491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1887.747934] LustreError: 50491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1887.804123] LustreError: 53070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788928111 with bad export cookie 17302987034850042519 [ 1887.805402] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1887.817171] LustreError: 53070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1888.253448] Lustre: server umount lustre-MDT0001 complete [ 1902.401367] Lustre: server umount lustre-OST0000 complete [ 1934.841538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.330585] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1935.400138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1935.405420] Lustre: Skipped 36 previous similar messages [ 1935.723606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.391439] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1951.980490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1952.215620] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1952.246723] Lustre: 64393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1957.717463] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1967.250642] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1973.381723] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1981.644581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1981.925177] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1982.294441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.299623] Lustre: Skipped 1 previous similar message [ 1989.714838] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1997.937632] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1999.906525] Lustre: 66962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2011.629376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2011.634350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.635370] LustreError: Skipped 1 previous similar message [ 2011.636205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2011.643498] Lustre: Skipped 2 previous similar messages [ 2011.654927] Lustre: Skipped 1 previous similar message [ 2014.317307] Lustre: server umount lustre-OST0000 complete [ 2019.315116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2021.867836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.879796] Lustre: Skipped 1 previous similar message [ 2025.182921] Lustre: server umount lustre-MDT0000 complete [ 2026.980439] LustreError: 66264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.006982] LustreError: 66264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2030.937799] LustreError: 64384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788928254 with bad export cookie 17302987034850045739 [ 2030.955341] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2030.957286] LustreError: 64384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2031.383517] Lustre: server umount lustre-MDT0001 complete [ 2047.117637] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 2050.347204] Key type lgssc unregistered [ 2050.699573] LNet: 68181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.714039] LNetError: 68181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.748684] LNet: Removed LNI 192.168.202.149@tcp [ 2051.725544] Key type .llcrypt unregistered [ 2051.727713] Key type ._llcrypt unregistered [ 2080.878739] Key type ._llcrypt registered [ 2080.881187] Key type .llcrypt registered [ 2081.007246] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2082.423068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2082.634042] alg: No test for adler32 (adler32-zlib) [ 2083.620795] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 2083.972502] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 2085.688148] Key type lgssc registered [ 2087.204965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2095.571205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2095.594971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.254441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.236775] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2107.365142] LustreError: 69597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.394302] LustreError: 69597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2110.426387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.173315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2116.351241] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2124.538885] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2131.714781] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2139.619142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2140.863325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2148.774876] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2159.054961] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2168.540646] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2181.094415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.103284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2181.129648] Lustre: Skipped 2 previous similar messages [ 2181.143576] Lustre: Skipped 2 previous similar messages [ 2185.208401] Lustre: server umount lustre-MDT0000 complete [ 2186.209829] LustreError: 69597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2186.242646] LustreError: 69597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2190.373829] LustreError: 69576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788928414 with bad export cookie 17987472792527013065 [ 2190.376467] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2190.401888] LustreError: 69576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2191.000212] Lustre: server umount lustre-MDT0001 complete [ 2206.224491] Lustre: server umount lustre-OST0000 complete [ 2240.552265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.886090] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2240.948086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2241.330291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2246.996175] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2255.476328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2255.559535] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2255.609827] Lustre: 74475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2255.977243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2261.116279] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2271.269704] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2279.189691] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2288.177231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2288.415859] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2288.863644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.526336] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2304.210899] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2305.984947] Lustre: 77050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2307.697970] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2316.257856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2316.264902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.274562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2320.411770] Lustre: server umount lustre-OST0000 complete [ 2326.506421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2326.522186] Lustre: Skipped 1 previous similar message [ 2326.532536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2326.539387] Lustre: Skipped 2 previous similar messages [ 2331.622131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2331.632586] Lustre: Skipped 1 previous similar message [ 2331.945104] Lustre: server umount lustre-MDT0000 complete [ 2335.893209] LustreError: 74465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788928559 with bad export cookie 17987472792527013905 [ 2335.896936] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2335.903831] LustreError: 74465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2336.124718] Lustre: server umount lustre-MDT0001 complete [ 2348.994962] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 2352.782914] Key type lgssc unregistered [ 2353.194149] LNet: 78261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2353.230878] LNetError: 78261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2353.261236] LNet: Removed LNI 192.168.202.149@tcp [ 2354.012830] Key type .llcrypt unregistered [ 2354.014656] Key type ._llcrypt unregistered [ 2385.426727] Key type ._llcrypt registered [ 2385.431498] Key type .llcrypt registered [ 2385.602228] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2386.611365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2386.713130] alg: No test for adler32 (adler32-zlib) [ 2387.781823] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 2387.965093] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 2389.624281] Key type lgssc registered [ 2391.104738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2401.570676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2401.604401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.279374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2408.258363] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2413.544879] LustreError: 79678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.563215] LustreError: 79678:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2418.666720] LustreError: 79679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.049104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2419.832119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2425.277751] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2434.975920] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2443.427994] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2453.148949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2453.982902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2462.339376] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2470.808714] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2482.395452] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2483.500282] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2484.929323] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2487.355180] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2487.363721] Lustre: Skipped 1 previous similar message [ 2491.484902] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2491.490387] Lustre: Skipped 2 previous similar messages [ 2500.217590] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2500.220382] Lustre: Skipped 7 previous similar messages [ 2516.961045] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2516.968635] Lustre: Skipped 14 previous similar messages [ 2549.321300] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2549.323568] Lustre: Skipped 30 previous similar messages [ 2605.167688] Lustre: 87791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2609.284068] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 00:40:30 (1788928830) [ 2612.023398] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2615.023957] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 00:40:36 (1788928836) [ 2621.921755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2621.929620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.941263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2621.943661] Lustre: Skipped 1 previous similar message [ 2621.974589] Lustre: Skipped 1 previous similar message [ 2626.379641] Lustre: server umount lustre-OST0000 complete [ 2632.163742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2632.165749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2632.179053] Lustre: Skipped 1 previous similar message [ 2632.189754] Lustre: Skipped 1 previous similar message [ 2637.282205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.288117] Lustre: Skipped 1 previous similar message [ 2637.669619] Lustre: server umount lustre-MDT0000 complete [ 2641.384396] LustreError: 79678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.401457] LustreError: 79678:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2641.705452] LustreError: 79660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788928865 with bad export cookie 17400296530476055202 [ 2641.707895] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2641.716740] LustreError: 79660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2641.985564] Lustre: server umount lustre-MDT0001 complete [ 2653.988355] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 2657.339145] Key type lgssc unregistered [ 2657.635824] LNet: 89045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2657.642302] LNetError: 89045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2657.674559] LNet: Removed LNI 192.168.202.149@tcp [ 2658.469195] Key type .llcrypt unregistered [ 2658.470328] Key type ._llcrypt unregistered [ 2663.806949] Key type ._llcrypt registered [ 2663.813805] Key type .llcrypt registered [ 2663.948436] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2669.674737] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules [ 2671.400109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2671.543519] alg: No test for adler32 (adler32-zlib) [ 2673.159815] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 2673.549214] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 2675.352139] Key type lgssc registered [ 2676.714161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2678.747268] LustreError: 90272:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2695.132704] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2700.837337] LustreError: 90272:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2708.864258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2708.919410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2710.863545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.911936] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2720.271462] LustreError: 91264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.301106] LustreError: 91264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2725.349811] LustreError: 91263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.586433] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 00:42:29 (1788928949) [ 2730.468277] LustreError: 91264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.024348] LustreError: 91295:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2734.036565] LustreError: 91295:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2734.044523] LustreError: 91295:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2734.417902] Lustre: server umount lustre-MDT0000 complete [ 2755.101942] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 2758.650608] Key type lgssc unregistered [ 2758.974206] LNet: 92793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2758.999395] LNetError: 92793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2759.027090] LNet: Removed LNI 192.168.202.149@tcp [ 2759.694971] Key type .llcrypt unregistered [ 2759.697363] Key type ._llcrypt unregistered [ 2780.953277] Key type ._llcrypt registered [ 2780.956016] Key type .llcrypt registered [ 2781.077890] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2782.407486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2782.588784] alg: No test for adler32 (adler32-zlib) [ 2783.794290] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 2784.180137] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 2785.896267] Key type lgssc registered [ 2787.746805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2797.948846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2797.989154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.011137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2805.696658] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2809.339128] LustreError: 94019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.372509] LustreError: 94019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2814.436708] LustreError: 94018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.755677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.791732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2824.789967] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2834.948371] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2842.342618] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2849.837873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2850.567461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2858.363806] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2866.687951] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2924.977558] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 00:45:46 (1788929146) [ 2926.800833] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2929.406540] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 00:45:50 (1788929150) [ 2936.804286] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2936.830033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2936.845370] Lustre: Skipped 2 previous similar messages [ 2941.938297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2941.942929] Lustre: Skipped 4 previous similar messages [ 2942.077246] Lustre: server umount lustre-MDT0000 complete [ 2945.833533] LustreError: 93997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788929169 with bad export cookie 5414070784401576482 [ 2945.839790] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2945.840888] LustreError: 93997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2946.262311] Lustre: server umount lustre-MDT0001 complete [ 2962.159661] Lustre: server umount lustre-OST0000 complete [ 2975.006115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.557102] LustreError: 98318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.730626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2980.343533] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2980.834281] LustreError: 98319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.960271] LustreError: 98318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2989.145166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2989.891199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2995.362981] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3003.475341] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3010.213116] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3024.012363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3024.314402] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3024.319240] LustreError: 98306:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3024.355208] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3024.364321] LustreError: 100402:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3024.374596] LustreError: 100402:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3024.381766] LustreError: 100402:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3024.484415] Lustre: server umount lustre-OST0000 complete [ 3024.486689] LustreError: 100402:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3034.787487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3034.988750] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3034.995603] LustreError: 98306:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3035.034903] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3035.058659] LustreError: 100868:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3035.076687] LustreError: 100873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.080921] LustreError: 100868:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3035.097833] LustreError: 100868:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3035.160794] Lustre: server umount lustre-OST0000 complete [ 3035.163695] LustreError: 100868:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3047.031662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3047.466331] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3047.481099] Lustre: Found index 0 for lustre-OST0000, updating log [ 3047.491033] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3047.590502] Lustre: lustre-OST0000: new disk, initializing [ 3047.595814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3047.601269] Lustre: 101403:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3047.921091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3055.238442] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3059.593377] Lustre: Failing over lustre-OST0000 [ 3059.705784] Lustre: server umount lustre-OST0000 complete [ 3066.215773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3066.223804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3066.236094] Lustre: Skipped 1 previous similar message [ 3066.248173] Lustre: Skipped 1 previous similar message [ 3071.454270] Lustre: server umount lustre-MDT0000 complete [ 3071.457082] LustreError: 98319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3071.463557] LustreError: 98319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3075.851857] LustreError: 98299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788929299 with bad export cookie 5414070784401579226 [ 3075.859056] LustreError: 98299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3075.863683] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3076.318857] Lustre: server umount lustre-MDT0001 complete [ 3085.140123] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 00:48:27 (1788929307) [ 3103.445869] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3117.514260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.011440] LustreError: 103749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.136760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.075912] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3133.342374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3139.923872] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3143.625795] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3151.714245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3152.567639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3152.572509] Lustre: Skipped 1 previous similar message [ 3159.588695] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3169.648102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3169.797746] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3169.943683] Lustre: lustre-OST0001: new disk, initializing [ 3169.948750] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3169.962654] Lustre: 106069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3178.266790] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3180.188641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3180.200654] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3180.381706] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3187.295840] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3197.440408] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3206.120401] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.121462] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3206.123131] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3206.137071] Lustre: Skipped 1 previous similar message [ 3208.711736] Lustre: server umount lustre-MDT0001 complete [ 3211.236902] LustreError: 103750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3211.260505] LustreError: 103750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3219.285046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.450792] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3219.461186] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3219.573698] Lustre: 103740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3220.215718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3220.229664] Lustre: Skipped 1 previous similar message [ 3221.575598] LustreError: 107625:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3221.585069] LustreError: 107625:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3221.592402] LustreError: 103743:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3221.730471] LustreError: 107644:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.149@tcp: cfg command failed: rc = -17 [ 3221.736995] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3221.736995] [ 3225.598776] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3225.758190] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3237.623076] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 00:50:59 (1788929459) [ 3257.899661] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3278.789754] Lustre: 109684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3305.246578] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3309.639282] Lustre: 110823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3399.650386] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.663171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3399.665487] Lustre: Skipped 3 previous similar messages [ 3399.680666] Lustre: Skipped 5 previous similar messages [ 3402.054109] Lustre: server umount lustre-MDT0000 complete [ 3404.271730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3404.282220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3404.297557] LustreError: 108146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.316507] LustreError: 108146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3407.252316] LustreError: 106067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788929631 with bad export cookie 5414070784401580220 [ 3407.275544] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3407.279548] LustreError: 106067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3408.027691] Lustre: server umount lustre-MDT0001 complete [ 3423.252568] Lustre: server umount lustre-OST0000 complete [ 3425.248701] Lustre: 93330:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788929633/real 1788929633] req@ffff8df54ecfd880 x1875828023867648/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788929649 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3425.284558] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3428.176656] Lustre: server umount lustre-OST0001 complete [ 3435.202420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3448.872707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3459.061386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3471.594593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3504.333787] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3517.259532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3517.290285] Lustre: lustre-MDT0000: reset Object Index mappings [ 3517.882320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3523.793910] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3534.948106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3534.972555] Lustre: lustre-MDT0001: reset Object Index mappings [ 3542.408891] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3546.287701] Lustre: 115948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3554.594898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3560.756915] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3562.809296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 3562.995935] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3573.291788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3579.373050] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3579.382613] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3581.617947] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3590.137410] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3594.155757] Lustre: 117824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3651.754981] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:57:53 (1788929873) [ 3676.647183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3676.664920] Lustre: Skipped 1 previous similar message [ 3676.678215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3682.371175] Lustre: server umount lustre-MDT0000 complete [ 3686.720327] LustreError: 115950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788929910 with bad export cookie 5414070784401627092 [ 3686.729844] LustreError: 115950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3686.743133] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3686.888684] LustreError: 114829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3686.900051] LustreError: 114829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 3695.012900] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3695.017404] Lustre: Skipped 12 previous similar messages [ 3701.218275] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3701.668652] Lustre: server umount lustre-MDT0001 complete [ 3713.344472] Lustre: server umount lustre-OST0000 complete [ 3725.098129] Lustre: server umount lustre-OST0001 complete [ 3731.794431] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 3746.052877] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3795.099667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3795.536126] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3795.572849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3795.728650] Lustre: lustre-MDT0000: new disk, initializing [ 3795.841267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3795.854435] Lustre: Skipped 3 previous similar messages [ 3795.880942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3801.650894] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3810.022403] Lustre: server umount lustre-MDT0000 complete [ 3821.293714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3821.636655] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3821.653582] Lustre: 122097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3821.672596] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3826.665780] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3832.051257] Lustre: server umount lustre-MDT0000 complete [ 3842.956697] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:01:04 (1788930064) [ 3845.075758] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3847.231487] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:01:09 (1788930069) [ 3866.177261] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3878.435202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3884.622819] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3895.213770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3895.372063] Lustre: 124156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3895.428941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3895.432961] Lustre: Skipped 1 previous similar message [ 3895.502838] Lustre: lustre-MDT0001: new disk, initializing [ 3895.654803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3895.667866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3901.235483] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3906.893731] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3915.165217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3915.594872] Lustre: lustre-OST0000: new disk, initializing [ 3915.600204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3915.609209] Lustre: 125736:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3917.211376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3917.241445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3917.506597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3924.376913] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3936.923323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3937.050392] Lustre: lustre-OST0001: new disk, initializing [ 3937.053765] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3937.059174] Lustre: 126606:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3938.849706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3938.860214] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3938.992523] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3944.817634] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3957.295416] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3961.912947] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4401.612815] Lustre: 133031:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4401.621633] Lustre: 133031:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4401.634427] Lustre: 133031:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4401.647470] Lustre: 133031:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4401.660551] Lustre: 133031:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4401.671498] Lustre: 133031:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4724.389643] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:15:45 (1788930945) [ 4743.148255] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.158241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4743.172127] Lustre: Skipped 7 previous similar messages [ 4743.188087] Lustre: Skipped 5 previous similar messages [ 4747.551784] Lustre: server umount lustre-MDT0000 complete [ 4748.265651] LustreError: 124165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4748.292213] LustreError: 124165:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 4751.933532] LustreError: 124151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788930975 with bad export cookie 5414070784401696448 [ 4751.936139] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4751.953991] LustreError: 124151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4753.382110] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4753.398236] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4753.412749] Lustre: Skipped 1 previous similar message [ 4753.434826] Lustre: Skipped 1 previous similar message [ 4758.797859] Lustre: server umount lustre-MDT0001 complete [ 4770.432614] Lustre: server umount lustre-OST0000 complete [ 4789.216251] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4789.617984] Lustre: server umount lustre-OST0001 complete [ 4799.200231] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 4813.309455] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 4864.582338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4865.061320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4865.102460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4865.264585] Lustre: lustre-MDT0000: new disk, initializing [ 4865.532532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4865.547076] Lustre: Skipped 5 previous similar messages [ 4865.577931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4870.786789] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4883.057509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4883.250859] Lustre: 136095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4883.319960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4883.324619] Lustre: Skipped 1 previous similar message [ 4883.406675] Lustre: lustre-MDT0001: new disk, initializing [ 4883.553814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4883.569869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4889.273668] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4899.543694] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4907.268984] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4916.976626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4917.454900] Lustre: lustre-OST0000: new disk, initializing [ 4917.464594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4917.476895] Lustre: 138063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4918.713493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4918.730047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4918.857913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4926.005549] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4936.851466] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4949.645322] Lustre: lustre-MDD0000: changelog on [ 4953.674537] Lustre: lustre-MDD0001: changelog on [ 7034.004253] Lustre: lustre-MDD0001: changelog off [ 7035.832843] Lustre: 139823:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 7036.616970] Lustre: lustre-MDD0000: changelog off [ 7046.629382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.647474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7046.657407] Lustre: Skipped 3 previous similar messages [ 7050.986683] Lustre: server umount lustre-OST0000 complete [ 7059.427616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7059.443630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7059.458667] Lustre: Skipped 1 previous similar message [ 7059.466843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7062.004270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7062.011946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7063.818599] Lustre: server umount lustre-MDT0000 complete [ 7066.594699] LustreError: 136971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7066.636237] LustreError: 136971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7070.142579] LustreError: 136088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933294 with bad export cookie 5414070784402516029 [ 7070.154510] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7070.155449] LustreError: 136088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7070.704057] Lustre: server umount lustre-MDT0001 complete [ 7085.505420] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 7090.874962] Key type lgssc unregistered [ 7091.695926] LNet: 147242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7091.714574] LNetError: 147242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7092.774828] LNet: Removed LNI 192.168.202.149@tcp [ 7094.938441] Key type .llcrypt unregistered [ 7094.942167] Key type ._llcrypt unregistered [ 7127.282150] Key type ._llcrypt registered [ 7127.291844] Key type .llcrypt registered [ 7127.555717] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 7146.642831] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7148.396035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7148.429202] alg: No test for adler32 (adler32-zlib) [ 7149.955082] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 7150.544777] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 7152.426334] Key type lgssc registered [ 7154.490892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7202.201547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7202.234494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7203.526232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7203.589459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7203.739446] Lustre: lustre-MDT0000: new disk, initializing [ 7203.795182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7203.808852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7209.349178] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7222.145875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7222.300499] Lustre: 150446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7222.372281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7222.375565] Lustre: Skipped 1 previous similar message [ 7222.441901] Lustre: lustre-MDT0001: new disk, initializing [ 7222.586742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7222.619154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7222.649416] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7227.919219] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7238.032769] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7245.648243] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7254.947832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7255.325757] Lustre: lustre-OST0000: new disk, initializing [ 7255.328594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7255.336662] Lustre: 152419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7255.415394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7262.422769] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7264.844456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7264.860758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7264.954749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7273.983663] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7280.350328] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_module obdecho/obdecho [ 7282.204822] LustreError: 153424:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7282.211885] LustreError: 153424:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7284.114874] LustreError: 153520:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7295.983382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7295.985569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7295.994893] Lustre: Skipped 2 previous similar messages [ 7296.011265] Lustre: Skipped 2 previous similar messages [ 7301.092345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7301.101458] Lustre: Skipped 2 previous similar messages [ 7302.257541] Lustre: server umount lustre-MDT0000 complete [ 7306.598350] LustreError: 150439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933530 with bad export cookie 1904626666432191708 [ 7306.600057] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7306.616840] LustreError: 150439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7307.007596] Lustre: server umount lustre-MDT0001 complete [ 7322.630455] Lustre: server umount lustre-OST0000 complete [ 7344.725996] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 7355.793903] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7403.173840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7403.542842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7403.583248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7403.709807] Lustre: lustre-MDT0000: new disk, initializing [ 7403.836357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7403.883090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7409.240992] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7422.367060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7422.524306] Lustre: 157242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7422.578726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7422.583322] Lustre: Skipped 1 previous similar message [ 7422.660825] Lustre: lustre-MDT0001: new disk, initializing [ 7422.755278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7422.802586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7422.821146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7429.332277] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7440.778894] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7447.857333] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7455.910423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7456.330657] Lustre: lustre-OST0000: new disk, initializing [ 7456.336864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7456.352926] Lustre: 159221:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7456.483292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7463.055505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7463.081136] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7463.199441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7464.343680] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7475.501063] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7480.947957] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7481.192624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7487.167414] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7487.478175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7494.117395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7494.143318] Lustre: Skipped 1 previous similar message [ 7494.152086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7494.154012] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7494.182081] LustreError: Skipped 1 previous similar message [ 7499.236046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7499.253479] Lustre: Skipped 1 previous similar message [ 7504.352854] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7504.372899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7504.393634] Lustre: Skipped 1 previous similar message [ 7504.558590] Lustre: server umount lustre-OST0000 complete [ 7509.985112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7509.997971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7514.593460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7514.594152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7514.604855] Lustre: Skipped 1 previous similar message [ 7516.157701] Lustre: server umount lustre-MDT0000 complete [ 7519.728847] LustreError: 158815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7519.778818] LustreError: 158815:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7520.963149] LustreError: 158037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933744 with bad export cookie 1904626666432192667 [ 7520.967643] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7520.981417] LustreError: 158037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7521.619306] Lustre: server umount lustre-MDT0001 complete [ 7525.894611] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:02:27 (1788933747) [ 7527.872630] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7529.657430] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:02:31 (1788933751) [ 7539.025731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7539.612650] LustreError: 161638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7539.654199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7544.805417] LustreError: 161639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7545.111961] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7549.922897] LustreError: 161638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7555.056239] LustreError: 161639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7556.269720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7563.241917] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7572.451356] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7580.496122] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7589.428539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7589.832213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7589.835247] Lustre: Skipped 1 previous similar message [ 7597.400757] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7606.531477] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7623.315318] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (stopping) [ 7623.329131] Lustre: Skipped 1 previous similar message [ 7623.657094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7623.676482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7625.545229] Lustre: server umount lustre-MDT0000 complete [ 7628.405193] LustreError: 162658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7629.536509] LustreError: 161618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933853 with bad export cookie 1904626666432193556 [ 7629.547147] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7629.550352] LustreError: 161618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7636.141854] Lustre: server umount lustre-MDT0001 complete [ 7644.397833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7659.752531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.901403] LustreError: 165506:0:(llog.c:1655:llog_backup()) MGC192.168.202.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7659.905908] Lustre: 165506:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7664.288489] LustreError: 165511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7664.319740] LustreError: 165511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7699.428079] LustreError: 165513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7699.460606] LustreError: 165513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 24 previous similar messages [ 7720.933743] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7720.941058] Lustre: 165506:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.149@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7761.948912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7765.600980] LustreError: 165511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7765.629402] LustreError: 165511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 7767.024133] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7769.203738] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7781.067030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7789.551562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7805.439856] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7806.743910] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7815.970685] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7817.724267] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7825.314287] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7827.108126] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7846.377407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7846.380111] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7846.393982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7846.403251] Lustre: Skipped 4 previous similar messages [ 7846.433308] Lustre: Skipped 8 previous similar messages [ 7852.376312] Lustre: server umount lustre-MDT0000 complete [ 7857.065184] LustreError: 161620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788934080 with bad export cookie 1904626666432200017 [ 7857.085815] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7857.089754] LustreError: 161620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7857.821482] Lustre: server umount lustre-MDT0001 complete [ 7873.099990] Lustre: server umount lustre-OST0000 complete [ 7899.452229] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 7911.041874] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7955.192908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7955.391803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7955.412139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7955.474817] Lustre: lustre-MDT0000: new disk, initializing [ 7955.543033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7955.549092] Lustre: Skipped 1 previous similar message [ 7955.563302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7961.329313] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7974.590433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7974.884761] Lustre: 170620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7974.980235] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7974.982902] Lustre: Skipped 3 previous similar messages [ 7975.366204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7980.700635] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7991.493607] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7997.468365] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8006.467867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8007.027443] Lustre: lustre-OST0000: new disk, initializing [ 8007.044496] Lustre: Skipped 1 previous similar message [ 8007.053916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8007.066670] Lustre: Skipped 2 previous similar messages [ 8007.076063] Lustre: 172586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8016.055684] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8016.965911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8016.979604] Lustre: Skipped 1 previous similar message [ 8016.993167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8017.216784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8027.521829] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8033.813142] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8034.088231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8040.959229] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8041.266602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8048.119194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8048.128292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8048.139893] LustreError: Skipped 1 previous similar message [ 8048.161491] Lustre: Skipped 2 previous similar messages [ 8049.717307] Lustre: server umount lustre-OST0000 complete [ 8068.066616] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8068.543326] Lustre: server umount lustre-MDT0000 complete [ 8068.578809] LustreError: 170624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8068.601677] LustreError: 170624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 34 previous similar messages [ 8072.389650] LustreError: 170610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788934296 with bad export cookie 1904626666432216852 [ 8072.395725] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8072.398830] LustreError: 170610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8072.826193] Lustre: server umount lustre-MDT0001 complete [ 8082.237733] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:11:44 (1788934304) [ 8090.515477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8096.415184] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8105.580711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8112.116419] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8123.874620] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8131.744408] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8140.602404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8150.635991] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8160.436986] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8173.564417] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:13:15 (1788934395) [ 8181.219829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8181.220833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8181.227206] LustreError: Skipped 2 previous similar messages [ 8181.233224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8181.242759] Lustre: Skipped 3 previous similar messages [ 8181.257946] Lustre: Skipped 8 previous similar messages [ 8183.842872] Lustre: server umount lustre-OST0000 complete [ 8194.434048] Lustre: server umount lustre-MDT0000 complete [ 8198.841189] LustreError: 174983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788934422 with bad export cookie 1904626666432217734 [ 8198.853838] LustreError: 174983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8198.857114] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8199.227359] Lustre: server umount lustre-MDT0001 complete [ 8212.157559] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 8215.305226] Key type lgssc unregistered [ 8215.653971] LNet: 178736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8215.661834] LNetError: 178736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8215.687975] LNet: Removed LNI 192.168.202.149@tcp [ 8216.665916] Key type .llcrypt unregistered [ 8216.669077] Key type ._llcrypt unregistered [ 8236.105661] Key type ._llcrypt registered [ 8236.110273] Key type .llcrypt registered [ 8236.238482] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8237.823837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8237.852932] alg: No test for adler32 (adler32-zlib) [ 8238.973176] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 8239.211684] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 8240.928168] Key type lgssc registered [ 8242.283351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8251.728641] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8251.751625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8267.360504] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8272.480385] LustreError: 179980:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.149@tcp: failed processing log, type 4: rc = -110 [ 8277.600513] LustreError: 179980:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8277.609209] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8277.615209] LustreError: MGC192.168.202.149@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8277.626300] LustreError: 179980:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8277.632251] LustreError: 179980:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8277.636791] LustreError: 179980:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8277.693656] Lustre: server umount lustre-OST0000 complete [ 8277.698798] LustreError: 179980:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8290.110824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8291.830081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8297.569033] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8302.050049] LustreError: 180480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8302.072846] LustreError: 180480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8307.120533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8307.182329] LustreError: 180481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8307.463567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8313.625688] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8322.606554] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8329.271729] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8336.946718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8337.152283] LustreError: 182297:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8337.155375] Lustre: 182297:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.149@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8337.353615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8344.801345] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8353.627276] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8357.876687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8357.883518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8357.905435] Lustre: Skipped 2 previous similar messages [ 8357.917047] Lustre: Skipped 2 previous similar messages [ 8362.838889] Lustre: server umount lustre-MDT0000 complete [ 8362.980844] LustreError: 181642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8363.005907] LustreError: 181642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8368.042962] LustreError: 180459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788934591 with bad export cookie 2342854329107911803 [ 8368.088758] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8368.124741] LustreError: 180477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8368.127118] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8368.181369] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8368.509609] Lustre: server umount lustre-MDT0001 complete [ 8375.566934] Lustre: server umount lustre-OST0000 complete [ 8387.704055] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 8399.207613] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8439.693837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8440.028468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8440.062546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8440.201875] Lustre: lustre-MDT0000: new disk, initializing [ 8440.292508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8440.312540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8445.400514] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8457.021413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8457.114710] Lustre: 185818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8457.160149] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8457.162280] Lustre: Skipped 1 previous similar message [ 8457.235568] Lustre: lustre-MDT0001: new disk, initializing [ 8457.350635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8457.379519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8457.393802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8462.589847] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8472.509068] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8479.664123] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8488.039022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8488.331536] Lustre: lustre-OST0000: new disk, initializing [ 8488.334611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8488.341628] Lustre: 187787:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8488.440100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8495.551861] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8498.286482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8498.298327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8498.383037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8506.215883] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8510.757429] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8510.965434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8516.233857] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8516.462864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8519.139269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8519.159236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8519.185098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8519.191975] Lustre: Skipped 1 previous similar message [ 8524.267480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8525.132207] Lustre: server umount lustre-OST0000 complete [ 8529.379118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8529.383419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8529.398960] Lustre: Skipped 2 previous similar messages [ 8529.414676] Lustre: Skipped 2 previous similar messages [ 8535.623613] Lustre: server umount lustre-MDT0000 complete [ 8539.618033] LustreError: 185831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8539.658156] LustreError: 185831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8539.785860] LustreError: 185813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788934763 with bad export cookie 2342854329107912510 [ 8539.813538] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8539.827885] LustreError: 185813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8540.672776] Lustre: server umount lustre-MDT0001 complete [ 8549.635448] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:19:31 (1788934771) [ 8569.799313] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 8572.651495] Key type lgssc unregistered [ 8573.003208] LNet: 190740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8573.009697] LNetError: 190740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8573.038598] LNet: Removed LNI 192.168.202.149@tcp [ 8574.057493] Key type .llcrypt unregistered [ 8574.062141] Key type ._llcrypt unregistered [ 8592.436073] Key type ._llcrypt registered [ 8592.439446] Key type .llcrypt registered [ 8592.584057] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8593.871274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8594.023252] alg: No test for adler32 (adler32-zlib) [ 8595.144953] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 8595.471870] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 8597.168271] Key type lgssc registered [ 8598.372995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8606.392253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8606.473124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8608.011588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8613.374460] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8617.445600] LustreError: 191962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8617.460907] LustreError: 191962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8622.555797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8622.563099] LustreError: 191961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8622.906756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8627.631891] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8638.944418] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8646.673510] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8656.983610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8657.178428] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8657.184793] LustreError: 193827:0:(llog.c:1696:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8657.196299] Lustre: 193827:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.149@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8657.370430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8665.902751] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8676.081064] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8679.403524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8679.421114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8679.440295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8682.978476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8682.978852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8682.999602] Lustre: Skipped 1 previous similar message [ 8683.017582] Lustre: Skipped 1 previous similar message [ 8688.106114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8688.118038] Lustre: Skipped 3 previous similar messages [ 8693.218826] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8693.237959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8693.250442] Lustre: Skipped 2 previous similar messages [ 8693.355409] Lustre: server umount lustre-MDT0000 complete [ 8697.767874] LustreError: 191942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788934921 with bad export cookie 16363807594345891085 [ 8697.773824] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8697.782350] LustreError: 191942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8698.234177] Lustre: server umount lustre-MDT0001 complete [ 8712.977275] Lustre: server umount lustre-OST0000 complete [ 8725.099462] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 8737.413737] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8779.494357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8779.750638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8779.785596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8779.857169] Lustre: lustre-MDT0000: new disk, initializing [ 8780.003739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8780.039719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8785.217508] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8796.922444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8797.059611] Lustre: 197354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8797.101924] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8797.106202] Lustre: Skipped 1 previous similar message [ 8797.108829] LustreError: 198136:0:(llog.c:1696:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8797.116512] LustreError: 198136:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8797.120701] Lustre: 198136:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8797.138692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8797.145364] Lustre: Skipped 1 previous similar message [ 8797.238679] Lustre: lustre-MDT0001: new disk, initializing [ 8797.346200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8797.380836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8797.392552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8802.493796] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8812.009424] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8817.623248] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8825.357996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8825.723934] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8825.726443] Lustre: Skipped 1 previous similar message [ 8825.731237] LustreError: 199299:0:(llog.c:1696:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8825.744544] LustreError: 199299:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8825.755672] Lustre: 199299:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8825.764877] Lustre: 199299:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8825.796488] Lustre: lustre-OST0000: new disk, initializing [ 8825.800912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8825.813181] Lustre: 199322:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8825.973862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8831.621620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8831.642059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8831.802178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8833.949745] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8843.888857] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8850.541152] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8850.726348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8856.769480] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8857.074915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8862.691765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8862.698300] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8862.699190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8862.721021] Lustre: Skipped 1 previous similar message [ 8865.980378] Lustre: server umount lustre-OST0000 complete [ 8872.940606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8872.973388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8872.973397] Lustre: Skipped 1 previous similar message [ 8872.999787] Lustre: Skipped 1 previous similar message [ 8878.715313] Lustre: server umount lustre-MDT0000 complete [ 8882.150912] LustreError: 200664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8882.176640] LustreError: 200664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8883.227438] LustreError: 198158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935107 with bad export cookie 16363807594345891806 [ 8883.235554] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8883.239018] LustreError: 198158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8883.799055] Lustre: server umount lustre-MDT0001 complete [ 8892.734677] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:25:14 (1788935114) [ 8910.781472] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8921.498589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8921.843276] LustreError: 202454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8921.891414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8926.826290] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8927.203485] LustreError: 202455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8936.425426] LustreError: 202454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8936.441730] LustreError: 202454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8936.602562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8941.078451] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8944.144713] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8950.224844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8956.584618] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8966.283992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8966.459484] Lustre: lustre-OST0001: new disk, initializing [ 8966.463161] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8966.468281] Lustre: 204760:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8966.559606] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8966.567083] Lustre: Skipped 2 previous similar messages [ 8973.465092] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8975.043584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8975.073394] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8975.209351] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8985.365340] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8995.426727] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8998.119049] Lustre: DEBUG MARKER: ADD OST3 [ 9003.684382] loop: module loaded [ 9003.749192] loop0: detected capacity change from 0 to 204800000 [ 9004.042570] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9004.056949] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9004.065015] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9011.744881] Lustre: DEBUG MARKER: STOP OST3 [ 9012.147224] loop0: detected capacity change from 0 to 204800000 [ 9019.439856] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9019.721215] Lustre: lustre-OST0003: new disk, initializing [ 9019.724782] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 9019.730239] Lustre: 206845:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9021.415943] Lustre: *** cfs_fail_loc=2109, val=2*** [ 9021.571235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 9021.577657] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 9021.605547] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 9022.263472] Lustre: Failing over lustre-OST0003 [ 9022.608955] Lustre: server umount lustre-OST0003 complete [ 9023.457264] LustreError: 206979:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 9026.546476] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9026.548577] LustreError: 203940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9026.555579] Lustre: Skipped 1 previous similar message [ 9026.573600] LustreError: 203940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9027.455113] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9039.149825] Lustre: DEBUG MARKER: START OST3 again [ 9047.070666] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9047.515366] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 9047.522932] Lustre: Skipped 1 previous similar message [ 9047.536766] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 9049.093591] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9049.531523] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9049.543446] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9054.955308] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9077.699046] Lustre: Failing over lustre-OST0003 [ 9077.880357] Lustre: server umount lustre-OST0003 complete [ 9078.254833] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9078.266935] LustreError: 203938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9078.279211] Lustre: Skipped 1 previous similar message [ 9078.305952] LustreError: 203938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 9098.728757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9098.734032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9098.737125] Lustre: Skipped 2 previous similar messages [ 9098.746764] Lustre: Skipped 6 previous similar messages [ 9102.542444] Lustre: server umount lustre-MDT0000 complete [ 9106.282486] LustreError: 202436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935330 with bad export cookie 16363807594345892688 [ 9106.304493] LustreError: 202436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9106.310298] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9112.644811] Lustre: server umount lustre-MDT0001 complete [ 9116.972433] Lustre: server umount lustre-OST0000 complete [ 9121.119525] Lustre: server umount lustre-OST0001 complete [ 9128.557852] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 9137.161919] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9173.311378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9173.527206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9173.569625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9173.663159] Lustre: lustre-MDT0000: new disk, initializing [ 9173.748395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9177.759081] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9188.809970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9188.900371] Lustre: 212373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9189.124459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9189.138503] Lustre: Skipped 1 previous similar message [ 9189.183602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9193.919280] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9203.051171] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9208.512231] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9216.081327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9216.325255] Lustre: lustre-OST0000: new disk, initializing [ 9216.327636] Lustre: Skipped 1 previous similar message [ 9216.330962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9216.334424] Lustre: Skipped 2 previous similar messages [ 9216.338496] Lustre: 214342:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9217.442728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9217.450619] Lustre: Skipped 1 previous similar message [ 9217.454204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9217.602633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9223.988671] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9232.967767] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9238.087880] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9238.279543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9242.875912] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9243.028951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9248.227863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9248.234558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9248.252671] Lustre: Skipped 3 previous similar messages [ 9248.276770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9248.288303] Lustre: Skipped 4 previous similar messages [ 9251.429112] Lustre: server umount lustre-OST0000 complete [ 9255.927267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9261.865470] Lustre: server umount lustre-MDT0000 complete [ 9263.588068] LustreError: 212385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9263.600524] LustreError: 212385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 9265.582180] LustreError: 212365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935489 with bad export cookie 16363807594345894984 [ 9265.582994] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9265.594368] LustreError: 212365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9266.211978] Lustre: server umount lustre-MDT0001 complete [ 9269.874982] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:31:31 (1788935491) [ 9287.617363] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 9301.775802] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9347.463702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9347.725429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9347.766364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9347.872751] Lustre: lustre-MDT0000: new disk, initializing [ 9348.013838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9353.482668] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9364.723848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9364.844407] Lustre: 218954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9365.092886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9370.743954] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9381.846561] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9388.408881] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9396.122505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9396.413399] Lustre: 220924:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9397.616884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9397.806032] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9403.613839] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9415.028229] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9421.939527] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9422.301223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9428.696947] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9428.946264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9433.572469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9433.584114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9433.599683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9433.603079] Lustre: Skipped 4 previous similar messages [ 9433.622097] Lustre: Skipped 4 previous similar messages [ 9445.344220] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9445.673815] Lustre: server umount lustre-OST0000 complete [ 9452.001518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9452.012420] LustreError: Skipped 1 previous similar message [ 9455.764802] Lustre: server umount lustre-MDT0000 complete [ 9459.170957] LustreError: 218965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9459.190250] LustreError: 218965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9460.262173] LustreError: 218945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935684 with bad export cookie 16363807594345895866 [ 9460.264230] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9460.285444] LustreError: 218945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9460.778982] Lustre: server umount lustre-MDT0001 complete [ 9470.568637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9471.433653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9471.443323] Lustre: Skipped 4 previous similar messages [ 9476.484052] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9485.691947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9491.269537] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9499.371679] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9505.174979] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9512.423443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9520.796038] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9528.740159] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9540.590302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9545.761515] Lustre: server umount lustre-OST0000 complete [ 9556.275851] Lustre: server umount lustre-MDT0000 complete [ 9560.027408] LustreError: 223126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935783 with bad export cookie 16363807594345896748 [ 9560.031488] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9560.037377] LustreError: 223126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9560.301537] Lustre: server umount lustre-MDT0001 complete [ 9667.338836] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 9671.203848] Key type lgssc unregistered [ 9671.812255] LNet: 226810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9671.824864] LNetError: 226810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9671.861909] LNet: Removed LNI 192.168.202.149@tcp [ 9673.172186] Key type .llcrypt unregistered [ 9673.174282] Key type ._llcrypt unregistered [ 9705.293286] Key type ._llcrypt registered [ 9705.301779] Key type .llcrypt registered [ 9705.557881] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 9727.881350] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9729.496106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9729.578479] alg: No test for adler32 (adler32-zlib) [ 9730.806341] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 9731.063402] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 9732.729419] Key type lgssc registered [ 9733.885700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9777.984798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9778.027312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9779.400295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9779.476855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9779.620929] Lustre: lustre-MDT0000: new disk, initializing [ 9779.764710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9779.834660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9785.796155] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9798.816569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9798.938592] Lustre: 230163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9799.010272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9799.013917] Lustre: Skipped 1 previous similar message [ 9799.115326] Lustre: lustre-MDT0001: new disk, initializing [ 9799.272963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9799.343427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9799.360477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9805.231971] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9815.397430] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9821.340939] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9829.284617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9829.730919] Lustre: lustre-OST0000: new disk, initializing [ 9829.742368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9829.767222] Lustre: 232135:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9829.991284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9835.127333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9835.136621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9835.299047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9838.270898] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9849.227313] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9855.027396] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9855.316694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9860.530940] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9860.721631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9866.216168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9866.219254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9866.241050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9866.251153] Lustre: Skipped 1 previous similar message [ 9868.999514] Lustre: server umount lustre-OST0000 complete [ 9875.938860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9875.944322] LustreError: Skipped 1 previous similar message [ 9875.947978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9875.968447] Lustre: Skipped 1 previous similar message [ 9875.976217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9879.214335] Lustre: server umount lustre-MDT0000 complete [ 9883.424049] LustreError: 230157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936107 with bad export cookie 698384332344182493 [ 9883.424868] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9883.443480] LustreError: 230157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9883.833343] Lustre: server umount lustre-MDT0001 complete [ 9906.436822] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9918.671198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9919.259932] LustreError: 235075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9919.295880] LustreError: 235075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9919.337597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9924.579606] LustreError: 235076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9925.050175] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9929.700409] LustreError: 235075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9934.821501] LustreError: 235076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9936.960119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9937.588854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9943.221298] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9946.914360] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9954.306527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9954.809905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9961.371365] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9972.109578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9972.548123] Lustre: lustre-OST0001: new disk, initializing [ 9972.580397] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9972.601664] Lustre: 237385:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9972.838643] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9979.476919] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9982.070542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9982.093162] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9982.236355] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9992.447949] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10001.742308] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10018.821250] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:44:00 (1788936240) [10023.403687] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10023.417025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10023.431219] Lustre: Skipped 4 previous similar messages [10023.433928] Lustre: Skipped 4 previous similar messages [10027.557954] Lustre: server umount lustre-MDT0000 complete [10028.517470] LustreError: 236579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10028.539768] LustreError: 236579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10032.113172] LustreError: 235058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936256 with bad export cookie 698384332344183375 [10032.114588] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10032.126638] LustreError: 235058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10032.548623] Lustre: server umount lustre-MDT0001 complete [10046.774546] Lustre: server umount lustre-OST0000 complete [10050.016740] Lustre: 228386:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788936257/real 1788936257] req@ffff8df54524ce00 x1875835307020416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788936273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10050.062470] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10053.097123] Lustre: server umount lustre-OST0001 complete [10061.569428] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [10073.519299] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10122.766267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10123.098895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10123.210319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10123.376126] Lustre: lustre-MDT0000: new disk, initializing [10123.508368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10123.539336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10128.759603] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10139.887094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10140.020729] Lustre: 241330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10140.093489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10140.096086] Lustre: Skipped 1 previous similar message [10140.148549] Lustre: lustre-MDT0001: new disk, initializing [10140.324428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10140.347244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10145.572094] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10155.475398] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10162.032796] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10169.700484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10169.957507] Lustre: lustre-OST0000: new disk, initializing [10169.960262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10169.964458] Lustre: 243301:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10171.964471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10171.981766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10172.144425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10177.538714] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10187.276118] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10193.561550] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10193.773402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10200.800497] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10201.199342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10208.237474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10208.241977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10208.262098] Lustre: Skipped 1 previous similar message [10208.267026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10208.284306] Lustre: Skipped 2 previous similar messages [10210.693944] Lustre: server umount lustre-OST0000 complete [10216.930850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10216.956378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10216.976039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10223.042499] Lustre: server umount lustre-MDT0000 complete [10223.586041] LustreError: 241342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10223.617462] LustreError: 241342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10227.635579] LustreError: 241323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936451 with bad export cookie 698384332344185510 [10227.637451] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10227.649256] LustreError: 241323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10228.198162] Lustre: server umount lustre-MDT0001 complete [10236.935607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10237.389143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10237.396199] Lustre: Skipped 2 previous similar messages [10242.529366] LustreError: 245520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10242.568159] LustreError: 245520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10243.896847] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10254.492032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10261.398489] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10271.619429] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10278.373409] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10287.612137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10296.141165] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10305.381017] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10453.477662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10453.491670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10453.512644] Lustre: Skipped 1 previous similar message [10453.516620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10453.521188] Lustre: Skipped 2 previous similar messages [10458.635679] Lustre: server umount lustre-OST0000 complete [10464.739264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10464.743204] LustreError: Skipped 1 previous similar message [10469.413301] Lustre: server umount lustre-MDT0000 complete [10471.909231] LustreError: 248027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10471.921429] LustreError: 248027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10473.838311] LustreError: 245500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936697 with bad export cookie 698384332344186392 [10473.845250] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10473.851854] LustreError: 245500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10474.607881] Lustre: server umount lustre-MDT0001 complete [10577.317427] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [10580.552231] Key type lgssc unregistered [10580.825678] LNet: 249138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10580.832445] LNetError: 249138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10581.877637] LNet: Removed LNI 192.168.202.149@tcp [10582.746190] Key type .llcrypt unregistered [10582.748961] Key type ._llcrypt unregistered [10614.603397] Key type ._llcrypt registered [10614.606832] Key type .llcrypt registered [10614.778714] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [10636.196153] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10637.886378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10637.897406] alg: No test for adler32 (adler32-zlib) [10639.046073] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [10639.347702] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [10641.088506] Key type lgssc registered [10642.265040] Lustre: Echo OBD driver; http://www.lustre.org/ [10702.128775] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10713.810293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10713.833491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10715.023646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10715.043940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10715.090606] Lustre: lustre-MDT0000: new disk, initializing [10715.190232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10715.199616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10719.575213] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10735.984195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.222498] Lustre: 253214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10736.286618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10736.290588] Lustre: Skipped 1 previous similar message [10736.423761] Lustre: lustre-MDT0001: new disk, initializing [10736.650363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10736.703963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10736.718862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10744.695557] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10750.281337] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10759.490158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10759.974539] Lustre: lustre-OST0000: new disk, initializing [10759.983939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10759.992870] Lustre: 254849:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10760.108305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10766.905796] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10768.977489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10768.994658] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10769.102530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10779.176856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10779.461536] Lustre: lustre-OST0001: new disk, initializing [10779.466428] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10779.474785] Lustre: 255722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10779.615071] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10785.431126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10785.434495] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10785.573525] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10788.727234] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10802.623413] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10811.986824] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10828.774707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10828.802781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10828.846708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10831.844361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10831.861990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10831.867102] Lustre: Skipped 1 previous similar message [10831.887937] Lustre: Skipped 1 previous similar message [10833.154556] Lustre: server umount lustre-MDT0000 complete [10838.122317] LustreError: 253209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937062 with bad export cookie 3657190034673595619 [10838.126944] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10838.148585] LustreError: 253209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10838.715851] Lustre: server umount lustre-MDT0001 complete [10853.445570] Lustre: server umount lustre-OST0000 complete [10857.445196] Lustre: 250720:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788937065/real 1788937065] req@ffff8df54acdb100 x1875836260143104/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788937081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10857.496974] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10857.514269] Lustre: Skipped 1 previous similar message [10857.974365] Lustre: server umount lustre-OST0001 complete [10877.692576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10889.388244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10898.766854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10914.442903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10940.959840] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10954.888997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10954.940045] Lustre: lustre-MDT0000: reset Object Index mappings [10955.675896] LustreError: 260737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10955.713971] LustreError: 260737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10955.801974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10960.868155] LustreError: 260738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10961.997664] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10965.987607] LustreError: 260737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10972.134970] LustreError: 260737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10972.165450] LustreError: 260737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10972.333875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10972.380445] Lustre: lustre-MDT0001: reset Object Index mappings [10972.932243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10978.806899] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10982.607876] Lustre: 261881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10990.718336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10990.956177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10997.555371] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10998.138500] LustreError: 262233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11001.298947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [11006.435076] LustreError: 262233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11006.483489] LustreError: 262233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11009.547947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11015.233105] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [11018.794106] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11030.260411] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11034.245916] Lustre: 263751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11039.718713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11039.727427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11039.751516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11039.764833] Lustre: Skipped 2 previous similar messages [11041.791494] Lustre: server umount lustre-MDT0000 complete [11045.461988] LustreError: 260719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937269 with bad export cookie 3657190034673602409 [11045.467601] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11045.488413] LustreError: 260719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11045.860832] LustreError: 260733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11045.870386] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11045.903721] LustreError: 260733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11045.956782] Lustre: Skipped 3 previous similar messages [11046.172856] Lustre: server umount lustre-MDT0001 complete [11051.840624] Lustre: server umount lustre-OST0000 complete [11055.148896] Lustre: server umount lustre-OST0001 complete [11080.117653] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:01:41 (1788937301) [11100.052163] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [11113.055434] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [11157.775202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11158.300421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11158.373873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11158.694164] Lustre: lustre-MDT0000: new disk, initializing [11158.816547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11158.825218] Lustre: Skipped 1 previous similar message [11158.840961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11166.739739] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11181.379278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11181.552894] Lustre: 268018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11181.610596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11181.613421] Lustre: Skipped 1 previous similar message [11181.657739] Lustre: lustre-MDT0001: new disk, initializing [11181.862675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11181.874800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11188.496417] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11202.240312] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11209.302601] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11217.842125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11218.172918] Lustre: lustre-OST0000: new disk, initializing [11218.175943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11218.184105] Lustre: 269988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11220.063214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11220.096475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11220.247739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11225.703345] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11236.664919] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11241.597061] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11241.814102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11246.826234] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11247.054650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11251.179703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11251.180303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11251.191258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11251.191268] Lustre: Skipped 4 previous similar messages [11251.199940] LustreError: Skipped 1 previous similar message [11251.221651] Lustre: Skipped 2 previous similar messages [11255.325380] Lustre: server umount lustre-OST0000 complete [11261.411298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11261.422097] Lustre: Skipped 2 previous similar messages [11265.869878] Lustre: server umount lustre-MDT0000 complete [11266.530846] LustreError: 268032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11266.552643] LustreError: 268032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11271.696418] LustreError: 268011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937495 with bad export cookie 3657190034673604502 [11271.724197] LustreError: 268011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11271.727335] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11272.407565] Lustre: server umount lustre-MDT0001 complete [11305.345317] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [11318.993093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11319.852313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11319.855089] Lustre: Skipped 2 previous similar messages [11327.375033] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11335.144335] LustreError: 272961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11335.173456] LustreError: 272961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11337.862270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11342.954124] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11346.445407] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11353.985478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11359.961664] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11371.217527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11371.419333] Lustre: lustre-OST0001: new disk, initializing [11371.435399] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11371.453362] Lustre: 275270:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11372.651841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11372.666911] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11372.808759] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11380.029897] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11392.128391] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11396.477574] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11401.619687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11401.682271] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11408.877434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11408.883262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11408.893987] Lustre: Skipped 4 previous similar messages [11408.909717] Lustre: Skipped 3 previous similar messages [11414.885865] Lustre: server umount lustre-MDT0000 complete [11418.611980] LustreError: 272940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937642 with bad export cookie 3657190034673605384 [11418.615606] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11418.640621] LustreError: 272940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11419.330273] Lustre: server umount lustre-MDT0001 complete [11433.941660] Lustre: server umount lustre-OST0000 complete [11434.336912] Lustre: 250723:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788937642/real 1788937642] req@ffff8df54c3fd180 x1875836260310144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788937658 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11439.332418] Lustre: 250721:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788937647/real 1788937647] req@ffff8df5531d7800 x1875836260310400/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788937663 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11440.260254] Lustre: server umount lustre-OST0001 complete [11461.574063] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [11472.518884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11472.770490] LustreError: 278327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11472.838102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11472.847144] Lustre: Skipped 3 previous similar messages [11479.050534] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11489.359362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11494.832707] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11498.437542] Lustre: 279467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11505.272445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11512.948905] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11519.117171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11522.892384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11528.176415] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11529.906767] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11538.730945] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11542.789768] Lustre: 281344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11548.114979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11548.161104] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11558.934462] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:09:40 (1788937780) [11574.242597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11574.249207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11574.252181] Lustre: Skipped 4 previous similar messages [11574.263895] Lustre: Skipped 7 previous similar messages [11576.110079] Lustre: server umount lustre-MDT0000 complete [11582.103610] LustreError: 281347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937805 with bad export cookie 3657190034673611922 [11582.111749] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11582.125369] LustreError: 281347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11588.850556] Lustre: server umount lustre-MDT0001 complete [11600.524482] Lustre: server umount lustre-OST0000 complete [11605.310836] Lustre: server umount lustre-OST0001 complete [11613.461940] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [11624.746275] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [11680.789893] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [11691.469723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11691.778286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11691.826445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11691.979431] Lustre: lustre-MDT0000: new disk, initializing [11692.128657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11697.780531] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11708.483089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11708.678026] Lustre: 285585:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11709.057598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11714.540737] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11728.774625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11729.154941] Lustre: 287219:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11729.273819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11729.280114] Lustre: Skipped 5 previous similar messages [11730.701297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11737.345949] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11750.759625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11751.018685] Lustre: 288090:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11753.095787] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11753.116070] Lustre: Skipped 1 previous similar message [11753.270139] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11758.929574] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11770.051752] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11774.156706] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11774.161605] Lustre: Skipped 1 previous similar message [11789.898488] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:13:31 (1788938011) [11794.403975] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11794.427619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11794.434726] Lustre: Skipped 6 previous similar messages [11794.441555] Lustre: Skipped 7 previous similar messages [11794.758163] Lustre: server umount lustre-MDT0000 complete [11798.668905] LustreError: 287218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938022 with bad export cookie 3657190034673619552 [11798.682917] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11798.690776] LustreError: 287218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11798.949568] Lustre: server umount lustre-MDT0001 complete [11812.319172] Lustre: server umount lustre-OST0000 complete [11815.264327] Lustre: 250722:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788938023/real 1788938023] req@ffff8df5493b3480 x1875836260463232/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788938039 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11816.552481] Lustre: server umount lustre-OST0001 complete [11826.032628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11826.378270] LustreError: 290340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11826.395210] LustreError: 290340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [11831.571913] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11842.197881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11849.027937] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11857.430522] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11863.945653] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11871.821219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11879.402669] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11888.197174] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11919.025314] loop1: detected capacity change from 0 to 204800000 [11919.370426] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11919.383598] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11919.397556] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11926.349472] loop1: detected capacity change from 0 to 204800000 [11935.453323] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11935.705211] LustreError: 290330:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs2736: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11935.737331] LustreError: 290330:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs2736 [11935.744666] LustreError: 290330:0:(mgs_handler.c:517:mgs_target_reg()) fs2736-MDT0001: Error checking target index: rc = -13 [11935.758846] LustreError: fs2736-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11935.768352] LustreError: 294084:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11935.773147] LustreError: 294084:0:(tgt_mount.c:2129:server_put_super()) no obd fs2736-MDT0001 [11935.779480] LustreError: 294084:0:(tgt_mount.c:116:server_deregister_mount()) fs2736-MDT0001 not registered [11935.833456] Lustre: server umount fs2736-MDT0001 complete [11935.835482] LustreError: 294084:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11947.958210] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11948.020664] Lustre: Setting parameter fs2736-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs2736-MDT0001 [11948.056686] Lustre: srv-fs2736-MDT0001: No data found on store. Initialize space. [11948.059896] Lustre: Skipped 4 previous similar messages [11948.135648] Lustre: fs2736-MDT0001: new disk, initializing [11948.137547] Lustre: Skipped 3 previous similar messages [11948.245036] Lustre: srv-fs2736-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11948.249647] Lustre: 294552:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2736-MDT0001-mdtlov: can't create new llog: rc = -115 [11948.253609] LustreError: 294552:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2736-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11954.585302] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11963.330387] Lustre: server umount fs2736-MDT0001 complete [11967.991159] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11973.117474] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11978.991389] loop1: detected capacity change from 0 to 204800000 [11985.236466] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11985.530332] Lustre: 296119:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2736-MDT0001-mdtlov: can't create new llog: rc = -115 [11985.538913] LustreError: 296119:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2736-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11990.444431] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11995.746634] Lustre: server umount fs2736-MDT0001 complete [12008.940882] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12008.943180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12008.959460] Lustre: Skipped 3 previous similar messages [12008.977051] Lustre: Skipped 2 previous similar messages [12013.405434] Lustre: server umount lustre-MDT0000 complete [12023.442148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12023.639217] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12029.432550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12029.500787] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12038.579283] loop1: detected capacity change from 0 to 204800000 [12044.815094] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12045.543352] Lustre: 298565:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2736-MDT0001-mdtlov: can't create new llog: rc = -115 [12045.552057] LustreError: 298565:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2736-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12050.915878] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12055.751887] Lustre: server umount fs2736-MDT0001 complete [12080.622635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12086.431799] Lustre: server umount lustre-OST0000 complete [12090.851714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12104.163402] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12104.571474] Lustre: server umount lustre-MDT0000 complete [12109.549572] LustreError: 291083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938333 with bad export cookie 3657190034673622233 [12109.556249] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12109.576516] LustreError: 291083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12110.000699] Lustre: server umount lustre-MDT0001 complete [12122.740495] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12125.827312] Key type lgssc unregistered [12126.206775] LNet: 300919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12126.228584] LNetError: 300919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12126.276641] LNet: Removed LNI 192.168.202.149@tcp [12127.203328] Key type .llcrypt unregistered [12127.205611] Key type ._llcrypt unregistered [12149.708635] Key type ._llcrypt registered [12149.710713] Key type .llcrypt registered [12149.849827] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [12151.181407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12151.261907] alg: No test for adler32 (adler32-zlib) [12152.392204] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [12152.742971] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [12154.427762] Key type lgssc registered [12155.500689] Lustre: Echo OBD driver; http://www.lustre.org/ [12164.029732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12164.106306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12165.533826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12172.035384] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12175.842113] LustreError: 302193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12175.875748] LustreError: 302193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12180.964864] LustreError: 302192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12183.028182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12183.678614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12188.719560] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12195.920221] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12203.297836] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12211.319260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12211.879033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12218.900549] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12227.881735] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12228.276857] LustreError: 304025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12233.707719] LustreError: 304024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12240.008353] LustreError: 304695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12240.032844] LustreError: 304695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12250.215491] LustreError: 304025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12250.223056] LustreError: 304025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12257.393484] loop1: detected capacity change from 0 to 204800000 [12257.611355] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12257.631314] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12257.653558] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12264.166686] loop1: detected capacity change from 0 to 204800000 [12266.479661] LustreError: 304024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12266.511784] LustreError: 304024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [12270.634208] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12270.832302] Lustre: fs20290-OST0000: new disk, initializing [12270.845320] Lustre: srv-fs20290-OST0000: No data found on store. Initialize space. [12270.862776] Lustre: 305951:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12270.966034] Lustre: fs20290-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12278.237670] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12287.492983] Lustre: server umount fs20290-OST0000 complete [12301.419851] LustreError: 304022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12301.441461] LustreError: 304022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [12302.129601] loop1: detected capacity change from 0 to 204800000 [12307.483232] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12308.240160] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12308.583359] Lustre: fs20290-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12312.555648] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12315.374411] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12325.918802] loop2: detected capacity change from 0 to 204800000 [12326.225540] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12326.246045] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12326.264852] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12333.079950] loop2: detected capacity change from 0 to 204800000 [12339.179101] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12339.266800] LustreError: fs20290-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12339.278115] LustreError: fs20290-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12339.288372] LustreError: 308990:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12339.293883] LustreError: 308990:0:(tgt_mount.c:2129:server_put_super()) no obd fs20290-OST0001 [12339.300058] LustreError: 308990:0:(tgt_mount.c:116:server_deregister_mount()) fs20290-OST0001 not registered [12339.349057] Lustre: server umount fs20290-OST0001 complete [12339.352311] LustreError: 308990:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12351.632925] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12351.800199] Lustre: fs20290-OST0001: new disk, initializing [12351.803093] Lustre: srv-fs20290-OST0001: No data found on store. Initialize space. [12351.811827] Lustre: 309542:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12351.893870] Lustre: fs20290-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12359.097499] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12368.003179] LustreError: 304022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12368.025293] LustreError: 304022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [12369.632769] Lustre: server umount fs20290-OST0000 complete [12380.980141] Lustre: server umount fs20290-OST0001 complete [12395.369156] loop1: detected capacity change from 0 to 204800000 [12401.786180] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12402.096498] Lustre: fs20290-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12408.154271] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12414.470194] loop2: detected capacity change from 0 to 204800000 [12422.543355] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12431.773992] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12444.146619] Lustre: server umount fs20290-OST0000 complete [12455.443448] Lustre: server umount fs20290-OST0001 complete [12461.037234] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12466.153733] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12471.268177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12471.272752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12471.276069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12471.284736] Lustre: Skipped 1 previous similar message [12471.307698] Lustre: Skipped 1 previous similar message [12475.599621] Lustre: server umount lustre-OST0000 complete [12480.481836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12480.499202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12480.530645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12481.516599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12481.538449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12481.541148] Lustre: Skipped 1 previous similar message [12486.030179] Lustre: server umount lustre-MDT0000 complete [12491.139972] LustreError: 302173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938715 with bad export cookie 2991264207987562948 [12491.141619] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12491.161988] LustreError: 302173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12491.449616] Lustre: server umount lustre-MDT0001 complete [12507.694266] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12511.593285] Key type lgssc unregistered [12511.864519] LNet: 315472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12511.871537] LNetError: 315472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12511.889138] LNet: Removed LNI 192.168.202.149@tcp [12513.309170] Key type .llcrypt unregistered [12513.313310] Key type ._llcrypt unregistered [12554.771864] Key type ._llcrypt registered [12554.774441] Key type .llcrypt registered [12554.920257] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [12571.874743] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [12573.710509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12573.765170] alg: No test for adler32 (adler32-zlib) [12574.966925] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [12575.292920] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [12576.944553] Key type lgssc registered [12578.296669] Lustre: Echo OBD driver; http://www.lustre.org/ [12625.071260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12625.112296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12626.460733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12626.492896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12626.593320] Lustre: lustre-MDT0000: new disk, initializing [12626.664203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12626.716367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12632.546695] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12644.704081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12644.915923] Lustre: 319338:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12645.020908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12645.035342] Lustre: Skipped 1 previous similar message [12645.172931] Lustre: lustre-MDT0001: new disk, initializing [12645.318993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12645.378817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12645.393090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12651.641885] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12663.710924] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12669.642861] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12676.465455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12676.636350] Lustre: lustre-OST0000: new disk, initializing [12676.641585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12676.645866] Lustre: 321307:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12676.706390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12682.671884] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12683.848217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12683.862486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12684.050573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12692.772228] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12695.387802] Lustre: Mounted lustre-client - version 2.17.57_55_ga2ebe9c [12696.625767] Lustre: Unmounted lustre-client [12709.862203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12709.866384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12709.881728] LustreError: Skipped 1 previous similar message [12709.892711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12709.911132] Lustre: Skipped 1 previous similar message [12710.898146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12710.920561] Lustre: Skipped 2 previous similar messages [12714.751504] Lustre: server umount lustre-OST0000 complete [12720.038042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12720.054268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12720.060156] Lustre: Skipped 1 previous similar message [12720.079055] Lustre: Skipped 1 previous similar message [12726.094764] Lustre: server umount lustre-MDT0000 complete [12730.665172] LustreError: 319330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938954 with bad export cookie 17992162808118842958 [12730.666676] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12730.678182] LustreError: 319330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12731.126382] Lustre: server umount lustre-MDT0001 complete [12745.166967] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12748.884380] Key type lgssc unregistered [12758.305232] LNet: 323471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12758.324178] LNetError: 323471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12825.964573] LNet: Removed LNI 192.168.202.149@tcp [12827.203588] Key type .llcrypt unregistered [12827.207179] Key type ._llcrypt unregistered [12849.207584] Key type ._llcrypt registered [12849.210549] Key type .llcrypt registered [12849.336762] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [12866.717907] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [12867.737531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12867.773967] alg: No test for adler32 (adler32-zlib) [12869.270401] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [12869.566826] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [12871.248253] Key type lgssc registered [12872.567080] Lustre: Echo OBD driver; http://www.lustre.org/ [12911.372727] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:32:12 (1788939132) [12919.923668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12919.956344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12921.289266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12921.317779] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12921.393442] Lustre: lustre-MDT0000: new disk, initializing [12921.466569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12921.486975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12927.147895] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12939.895781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12940.049628] Lustre: 326695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12940.113520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12940.118151] Lustre: Skipped 1 previous similar message [12940.214311] Lustre: lustre-MDT0001: new disk, initializing [12940.331677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12940.382807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12940.389793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12945.134665] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12956.525702] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12964.077342] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12971.684061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12972.032642] Lustre: lustre-OST0000: new disk, initializing [12972.036401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12972.042640] Lustre: 328664:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12972.142561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12979.094791] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12980.315108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12980.337691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12980.461527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12990.166796] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13006.306606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13006.319608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13006.339126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13011.436619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13011.450972] Lustre: Skipped 1 previous similar message [13012.212820] Lustre: server umount lustre-OST0000 complete [13021.669139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13021.678784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13021.679281] Lustre: Skipped 2 previous similar messages [13021.687876] Lustre: Skipped 2 previous similar messages [13023.654773] Lustre: server umount lustre-MDT0000 complete [13028.204752] LustreError: 326688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939252 with bad export cookie 162769215595633530 [13028.208808] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13028.229754] LustreError: 326688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13028.623756] Lustre: server umount lustre-MDT0001 complete [13042.700760] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [13046.829421] Key type lgssc unregistered [13047.351274] LNet: 330573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13047.370188] LNetError: 330573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13047.399166] LNet: Removed LNI 192.168.202.149@tcp [13048.961607] Key type .llcrypt unregistered [13048.973247] Key type ._llcrypt unregistered [13075.949070] Key type ._llcrypt registered [13075.956455] Key type .llcrypt registered [13076.175228] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [13077.661171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13077.709179] alg: No test for adler32 (adler32-zlib) [13078.869707] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [13079.137355] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [13080.800199] Key type lgssc registered [13082.349794] Lustre: Echo OBD driver; http://www.lustre.org/ [13091.445718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13091.479343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13093.117279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13097.309536] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13102.571203] LustreError: 331851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13102.595655] LustreError: 331851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13107.560957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13107.682107] LustreError: 331850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13108.003682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13112.132633] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13121.096978] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13126.409491] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13132.717102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13133.000550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13141.377738] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13150.948775] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13162.844889] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13164.995042] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13169.634335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13169.646374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13169.671045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13173.235444] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13173.248555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13173.252192] Lustre: Skipped 1 previous similar message [13175.696533] Lustre: server umount lustre-MDT0000 complete [13178.353519] LustreError: 331851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13178.382303] LustreError: 331851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13179.633814] LustreError: 331831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939403 with bad export cookie 4245997608243441608 [13179.660188] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13179.686589] LustreError: 331831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13183.457931] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13183.463048] LustreError: 331846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13183.470863] Lustre: Skipped 1 previous similar message [13183.472225] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13183.489887] LustreError: 331846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13183.505544] Lustre: Skipped 2 previous similar messages [13186.104307] Lustre: server umount lustre-MDT0001 complete [13191.430774] Lustre: server umount lustre-OST0000 complete [13202.312930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13208.596534] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13212.641586] Lustre: 336035:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.149@tcp [13215.508862] Lustre: server umount MGS complete [13226.099954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13226.835411] LustreError: 336607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13226.870802] LustreError: 336607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13227.022874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13234.399714] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13236.271773] LustreError: 336608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13236.288631] LustreError: 336608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13244.663268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13244.914919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13250.322125] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13262.074338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13262.640916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13270.504805] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13279.495614] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13281.546043] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13287.122040] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13288.613292] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13295.095259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [13295.107253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13295.148189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13295.162023] Lustre: Skipped 2 previous similar messages [13302.757732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13302.769101] Lustre: Skipped 4 previous similar messages [13308.384726] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13308.548423] Lustre: server umount lustre-OST0000 complete [13312.994129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13313.023141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13313.025770] Lustre: Skipped 2 previous similar messages [13313.037324] Lustre: Skipped 3 previous similar messages [13319.004617] Lustre: server umount lustre-MDT0000 complete [13323.136596] LustreError: 336587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939547 with bad export cookie 4245997608243442448 [13323.153043] LustreError: 336587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13323.154900] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13323.574456] Lustre: server umount lustre-MDT0001 complete [13337.631276] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [13340.578539] Key type lgssc unregistered [13340.985775] LNet: 340049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13340.998574] LNetError: 340049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13341.031955] LNet: Removed LNI 192.168.202.149@tcp [13341.863519] Key type .llcrypt unregistered [13341.865275] Key type ._llcrypt unregistered [13365.888668] Key type ._llcrypt registered [13365.893263] Key type .llcrypt registered [13366.034202] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [13386.888829] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [13389.656674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13389.759219] alg: No test for adler32 (adler32-zlib) [13390.860289] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [13391.161296] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [13393.072274] Key type lgssc registered [13395.023130] Lustre: Echo OBD driver; http://www.lustre.org/ [13446.826321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13446.880485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13448.395703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13448.456550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13448.590842] Lustre: lustre-MDT0000: new disk, initializing [13448.771361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13448.822515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13457.208743] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13473.605944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13473.705797] Lustre: 343215:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13473.754557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13473.758691] Lustre: Skipped 1 previous similar message [13473.812660] Lustre: lustre-MDT0001: new disk, initializing [13473.892192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13473.972606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13473.990444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13479.782669] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13492.035323] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13499.179831] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13507.229237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13507.631572] Lustre: lustre-OST0000: new disk, initializing [13507.635459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13507.640392] Lustre: 345187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13507.746907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13515.349815] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13517.977484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13517.995715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13518.200210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13526.425249] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13531.728652] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13532.091073] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13537.891419] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13538.252257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13543.912809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13543.919714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13543.938573] Lustre: Skipped 1 previous similar message [13543.939312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13546.989198] Lustre: server umount lustre-OST0000 complete [13554.149902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13554.176186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13554.184166] Lustre: Skipped 1 previous similar message [13554.211100] Lustre: Skipped 2 previous similar messages [13557.754622] Lustre: server umount lustre-MDT0000 complete [13562.107213] LustreError: 343208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939785 with bad export cookie 3161163102766309932 [13562.124420] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13562.125184] LustreError: 343208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13562.581596] Lustre: server umount lustre-MDT0001 complete [13572.495150] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 03:43:14 (1788939794) [13589.581935] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [13600.549914] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [13649.628909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13649.956053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13649.983425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13650.041573] Lustre: lustre-MDT0000: new disk, initializing [13650.095113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13650.113722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13654.292796] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13666.551749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13666.691211] Lustre: 349947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13666.764757] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13666.768958] Lustre: Skipped 1 previous similar message [13666.880763] Lustre: lustre-MDT0001: new disk, initializing [13666.977416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13667.033936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13667.043139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13671.577958] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13684.057583] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13692.369593] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13701.194656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13701.416590] Lustre: lustre-OST0000: new disk, initializing [13701.419350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13701.423696] Lustre: 351918:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13701.543881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13708.205700] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13711.443535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13711.458686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13711.687660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13719.228872] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13763.048192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13763.049579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13763.061616] LustreError: Skipped 2 previous similar messages [13763.080017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13763.082216] Lustre: Skipped 1 previous similar message [13768.165076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13768.182103] Lustre: Skipped 2 previous similar messages [13773.301073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13773.317817] Lustre: Skipped 1 previous similar message [13774.306437] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13774.459327] Lustre: server umount lustre-OST0000 complete [13779.938611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13779.953657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13783.528158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13783.535723] Lustre: Skipped 2 previous similar messages [13785.191775] Lustre: server umount lustre-MDT0000 complete [13788.646930] LustreError: 352739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13788.675022] LustreError: 352739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13789.177127] LustreError: 349939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940013 with bad export cookie 3161163102766310821 [13789.182570] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13789.186193] LustreError: 349939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [13789.569179] Lustre: server umount lustre-MDT0001 complete [13801.548339] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [13804.559749] Key type lgssc unregistered [13805.120837] LNet: 354100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13805.136251] LNetError: 354100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13806.184384] LNet: Removed LNI 192.168.202.149@tcp [13807.186918] Key type .llcrypt unregistered [13807.189879] Key type ._llcrypt unregistered [13833.926147] Key type ._llcrypt registered [13833.939942] Key type .llcrypt registered [13834.182970] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [13835.554667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13835.670867] alg: No test for adler32 (adler32-zlib) [13837.788521] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [13838.231395] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [13839.976196] Key type lgssc registered [13841.808593] Lustre: Echo OBD driver; http://www.lustre.org/ [13850.004158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13850.028673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13851.804706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13857.868986] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13861.347860] LustreError: 355488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13861.399749] LustreError: 355488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13866.468986] LustreError: 355489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13868.668477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13869.490104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13875.008682] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13882.565356] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13891.829736] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13903.370691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13903.764877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13911.562414] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13921.639384] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13941.217978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13941.229114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13941.248127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13941.749902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13941.758505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13941.772493] Lustre: Skipped 1 previous similar message [13941.785340] Lustre: Skipped 1 previous similar message [13946.770896] Lustre: server umount lustre-MDT0000 complete [13946.858560] LustreError: 355484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13946.891581] LustreError: 355484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13952.010150] LustreError: 355488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13952.043563] LustreError: 355488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13952.944077] LustreError: 355469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940176 with bad export cookie 3508336585298451561 [13952.949534] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13952.950772] LustreError: 355469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13953.248698] Lustre: server umount lustre-MDT0001 complete [13968.382856] Lustre: server umount lustre-OST0000 complete [14032.768956] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [14044.040484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14044.328045] LustreError: 360732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14044.417560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14051.097541] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [14055.007175] LustreError: 360733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14055.035907] LustreError: 360733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14067.662555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14068.479893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14075.169362] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [14078.720891] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14086.146993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14086.493348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14089.596286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [14092.712782] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [14102.279346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14102.564454] Lustre: lustre-OST0001: new disk, initializing [14102.578140] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [14102.594931] Lustre: 363043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14102.749904] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14111.363387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14111.390075] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14111.621539] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14112.218932] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [14124.816551] Lustre: DEBUG MARKER: Using TIMEOUT=20 [14135.659764] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [14138.589685] Lustre: DEBUG MARKER: == conf-sanity test 301: lfind --fid2path names a device scan's objects ========================================================== 03:52:39 (1788940359) [14146.749353] Lustre: Mounted lustre-client - version 2.17.57_55_ga2ebe9c [14151.415132] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [14168.339042] Lustre: Unmounted lustre-client [14171.379625] Lustre: DEBUG MARKER: == conf-sanity test 302: lfind --fid2path names objects on a stopped OST ========================================================== 03:53:12 (1788940392) [14177.575290] LustreError: 354797:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [14177.613741] Lustre: Mounted lustre-client - version 2.17.57_55_ga2ebe9c [14182.107244] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [14187.031844] Lustre: Failing over lustre-OST0000 [14187.182409] Lustre: server umount lustre-OST0000 complete [14188.018547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14188.036603] Lustre: Skipped 1 previous similar message [14188.036653] LustreError: 363367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14188.065848] LustreError: 363367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14206.333486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14206.573394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14206.595986] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [14207.755211] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 4 clients reconnect [14207.880373] Lustre: lustre-OST0000: Recovery over after 0:01, of 4 clients 4 recovered and 0 were evicted. [14207.884852] Lustre: lustre-OST0000-osc-ffff8df54f63c800: Connection restored to 0@lo (at 0@lo) [14214.420655] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [14221.060671] Lustre: Unmounted lustre-client [14223.957744] Lustre: DEBUG MARKER: == conf-sanity test 303: lfind --paths names an MDT with no mount at all ========================================================== 03:54:06 (1788940446) [14243.136276] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 03:54:24 (1788940464) [14252.115609] Lustre: server umount lustre-MDT0000 complete [14252.528192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14252.537810] LustreError: 360729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14252.549900] Lustre: Skipped 4 previous similar messages [14252.566895] LustreError: 360729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [14257.637721] LustreError: 360713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940481 with bad export cookie 3508336585298456181 [14257.667739] LustreError: 360713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14257.686480] LustreError: 354797:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8df544bc4700 x1875839614641024/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [14257.691210] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14257.692835] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14257.818141] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14258.490310] Lustre: server umount lustre-MDT0001 complete [14264.016133] Lustre: server umount lustre-OST0000 complete [14278.972099] Lustre: server umount lustre-OST0001 complete [14326.388308] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 03:55:47 (1788940547) [14375.287204] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [14378.953837] Key type lgssc unregistered [14379.302362] LNet: 370313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14379.315345] LNetError: 370313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14379.341631] LNet: Removed LNI 192.168.202.149@tcp [14380.573425] Key type .llcrypt unregistered [14380.578724] Key type ._llcrypt unregistered [14475.013259] Key type ._llcrypt registered [14475.016355] Key type .llcrypt registered [14475.131657] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [14475.536409] Key type .llcrypt unregistered [14475.542344] Key type ._llcrypt unregistered [14508.975495] Key type ._llcrypt registered [14508.977232] Key type .llcrypt registered [14509.086155] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [14509.397449] Key type .llcrypt unregistered [14509.399654] Key type ._llcrypt unregistered [14548.785532] Key type ._llcrypt registered [14548.787070] Key type .llcrypt registered [14548.884196] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [14549.246472] Key type .llcrypt unregistered [14549.249771] Key type ._llcrypt unregistered [14634.233375] Key type ._llcrypt registered [14634.235814] Key type .llcrypt registered [14634.352444] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [14634.719464] Key type .llcrypt unregistered [14634.721391] Key type ._llcrypt unregistered [14668.345185] Key type ._llcrypt registered [14668.349293] Key type .llcrypt registered [14668.520858] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [14668.846605] Key type .llcrypt unregistered [14668.849427] Key type ._llcrypt unregistered [14705.049118] Key type ._llcrypt registered [14705.051760] Key type .llcrypt registered [14705.255549] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [14705.660453] Key type .llcrypt unregistered [14705.664213] Key type ._llcrypt unregistered [14734.851633] Key type ._llcrypt registered [14734.858928] Key type .llcrypt registered [14735.005558] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [14735.339510] Key type .llcrypt unregistered [14735.341620] Key type ._llcrypt unregistered [14770.742759] Key type ._llcrypt registered [14770.749462] Key type .llcrypt registered [14770.945039] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [14771.214605] Key type .llcrypt unregistered [14771.216115] Key type ._llcrypt unregistered [14811.177897] Key type ._llcrypt registered [14811.180600] Key type .llcrypt registered [14811.294192] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [14811.599631] Key type .llcrypt unregistered [14811.601667] Key type ._llcrypt unregistered [14875.826143] Key type ._llcrypt registered [14875.828738] Key type .llcrypt registered [14875.947969] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [14876.216447] Key type .llcrypt unregistered [14876.218692] Key type ._llcrypt unregistered [14912.688635] Key type ._llcrypt registered [14912.690815] Key type .llcrypt registered [14912.857640] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [14913.185487] Key type .llcrypt unregistered [14913.187158] Key type ._llcrypt unregistered [14949.261631] Key type ._llcrypt registered [14949.265446] Key type .llcrypt registered [14949.429460] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [14949.781540] Key type .llcrypt unregistered [14949.783107] Key type ._llcrypt unregistered [15007.020603] Key type ._llcrypt registered [15007.023494] Key type .llcrypt registered [15007.132786] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [15022.869642] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [15024.013651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15024.049386] alg: No test for adler32 (adler32-zlib) [15025.093137] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [15025.327677] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [15026.968238] Key type lgssc registered [15028.291686] Lustre: Echo OBD driver; http://www.lustre.org/ [15065.078818] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15065.105908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15066.279959] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [15066.298059] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [15066.360977] Lustre: tst250fs-MDT0000: new disk, initializing [15066.427426] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15066.450523] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15070.286128] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [15085.235333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15085.451927] Lustre: 387859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15085.564151] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [15085.581756] Lustre: Skipped 1 previous similar message [15085.718140] Lustre: tst250fs-MDT0001: new disk, initializing [15085.965838] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15086.064747] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15086.076566] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15093.529503] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [15099.660818] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [15109.123843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15109.538420] Lustre: tst250fs-OST0000: new disk, initializing [15109.544525] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [15109.552767] Lustre: 389493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15109.679102] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15116.383229] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [15116.882773] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15116.889831] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15117.021689] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15127.268495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15127.463473] Lustre: tst250fs-OST0001: new disk, initializing [15127.467338] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [15127.472664] Lustre: 390367:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15127.578595] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [15135.530731] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [15136.394215] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [15136.421336] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [15136.557837] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15141.823665] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [15144.954922] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [15144.962407] Lustre: Skipped 2 previous similar messages [15167.463739] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15167.478440] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [15167.486322] Lustre: Skipped 2 previous similar messages [15170.561688] Lustre: server umount tst250fs-MDT0000 complete [15174.440582] LustreError: 387851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941398 with bad export cookie 6967389223193860006 [15174.444820] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15174.447852] LustreError: 387851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15174.894956] Lustre: server umount tst250fs-MDT0001 complete [15188.746171] Lustre: server umount tst250fs-OST0000 complete [15202.954576] Lustre: server umount tst250fs-OST0001 complete [15226.226507] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [15235.106446] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [15277.479713] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:11:39 (1788941499) [15279.740395] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [15298.464771] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [15309.084668] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [15346.105450] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15076 sec ========= 04:12:47 (1788941567) [15348.721956] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:12:50 (1788941570) === [15353.448957] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:12:55 (1788941575) === [15397.861708] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [15401.435154] Key type lgssc unregistered [15401.751507] LNet: 399524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15401.758138] LNetError: 399524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15401.780138] LNet: Removed LNI 192.168.202.149@tcp [15403.182171] Key type .llcrypt unregistered [15403.188385] Key type ._llcrypt unregistered