[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 493681596 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003297] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012047] SELinux: Initializing. [ 0.013073] *** VALIDATE selinux *** [ 0.020705] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025193] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027093] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028144] *** VALIDATE tmpfs *** [ 0.030396] *** VALIDATE proc *** [ 0.032063] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.035206] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037038] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043029] debug: unmapping init [mem 0xffffffff97659000-0xffffffff97660fff] [ 0.045199] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046699] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047022] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051018] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054292] rcu: Hierarchical SRCU implementation. [ 0.056409] smp: Bringing up secondary CPUs ... [ 0.057497] x86: Booting SMP configuration: [ 0.058022] .... node #0, CPUs: #1 #2 #3 [ 0.061415] smp: Brought up 1 node, 4 CPUs [ 0.063011] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141017] node 0 deferred pages initialised in 74ms [ 0.144197] devtmpfs: initialized [ 0.145239] x86/mm: Memory block size: 128MB [ 0.148591] gcov: version magic: 0x41383552 [ 0.151213] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153267] pinctrl core: initialized pinctrl subsystem [ 0.154178] [ 0.154550] ************************************************************* [ 0.155012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156013] ** ** [ 0.157009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158013] ** ** [ 0.159014] ** This means that this kernel is built to expose internal ** [ 0.160012] ** IOMMU data structures, which may compromise security on ** [ 0.161011] ** your system. ** [ 0.162010] ** ** [ 0.163013] ** If you see this message and you are not debugging the ** [ 0.164013] ** kernel, report this immediately to your vendor! ** [ 0.165010] ** ** [ 0.166013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167014] ************************************************************* [ 0.168732] NET: Registered protocol family 16 [ 0.170413] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.178062] cpuidle: using governor menu [ 0.179754] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182401] PCI: Using configuration type 1 for base access [ 0.185136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193212] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.200032] cryptd: max_cpu_qlen set to 1000 [ 0.203218] ACPI: Added _OSI(Module Device) [ 0.205044] ACPI: Added _OSI(Processor Device) [ 0.207016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.209014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.213976] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220513] ACPI: Interpreter enabled [ 0.222065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.224013] ACPI: Using IOAPIC for interrupt routing [ 0.225118] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.230459] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.240416] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.243038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.245013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.248078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.253323] acpiphp: Slot [2] registered [ 0.255118] acpiphp: Slot [5] registered [ 0.257134] acpiphp: Slot [6] registered [ 0.258141] acpiphp: Slot [7] registered [ 0.260126] acpiphp: Slot [8] registered [ 0.262083] acpiphp: Slot [9] registered [ 0.263122] acpiphp: Slot [10] registered [ 0.265104] acpiphp: Slot [3] registered [ 0.267122] acpiphp: Slot [4] registered [ 0.268100] acpiphp: Slot [11] registered [ 0.270086] acpiphp: Slot [12] registered [ 0.271098] acpiphp: Slot [13] registered [ 0.273154] acpiphp: Slot [14] registered [ 0.274102] acpiphp: Slot [15] registered [ 0.276103] acpiphp: Slot [16] registered [ 0.277122] acpiphp: Slot [17] registered [ 0.279108] acpiphp: Slot [18] registered [ 0.280097] acpiphp: Slot [19] registered [ 0.282125] acpiphp: Slot [20] registered [ 0.284108] acpiphp: Slot [21] registered [ 0.285094] acpiphp: Slot [22] registered [ 0.287120] acpiphp: Slot [23] registered [ 0.288097] acpiphp: Slot [24] registered [ 0.290102] acpiphp: Slot [25] registered [ 0.291103] acpiphp: Slot [26] registered [ 0.292147] acpiphp: Slot [27] registered [ 0.294089] acpiphp: Slot [28] registered [ 0.295104] acpiphp: Slot [29] registered [ 0.297090] acpiphp: Slot [30] registered [ 0.298087] acpiphp: Slot [31] registered [ 0.299051] PCI host bridge to bus 0000:00 [ 0.300019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.302020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.305021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.307017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.309021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.311022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.313161] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.315972] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.319261] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.329014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.333527] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.336019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.338016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.340016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.344083] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.347793] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.350052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.354814] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.360014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.374026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.379766] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.386475] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.392015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.418016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.428278] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.434015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.440016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.455019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.467540] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.473011] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.481014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.495022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.504000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.513016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.520019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.542016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.551000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.567013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.577018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.601012] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.615570] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.629015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.643019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.665016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.677548] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.680365] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.683394] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.686387] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.689273] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.693245] iommu: Default domain type: Passthrough [ 0.695587] SCSI subsystem initialized [ 0.697157] ACPI: bus type USB registered [ 0.699122] usbcore: registered new interface driver usbfs [ 0.701131] usbcore: registered new interface driver hub [ 0.704083] usbcore: registered new device driver usb [ 0.706183] pps_core: LinuxPPS API ver. 1 registered [ 0.708011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.712062] PTP clock support registered [ 0.714080] EDAC MC: Ver: 3.0.0 [ 0.716128] PCI: Using ACPI for IRQ routing [ 0.718890] NetLabel: Initializing [ 0.720010] NetLabel: domain hash size = 128 [ 0.722011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.724124] NetLabel: unlabeled traffic allowed by default [ 0.727010] vgaarb: loaded [ 0.728342] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.731021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.737023] clocksource: Switched to clocksource kvm-clock [ 0.841547] VFS: Disk quotas dquot_6.6.0 [ 0.843740] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.846626] *** VALIDATE ramfs *** [ 0.848061] *** VALIDATE hugetlbfs *** [ 0.849751] pnp: PnP ACPI init [ 0.852446] pnp: PnP ACPI: found 6 devices [ 0.871435] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.874879] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.877256] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.879585] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.882150] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.884873] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.887919] NET: Registered protocol family 2 [ 0.890696] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.896500] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.900064] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.905831] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.909627] TCP: Hash tables configured (established 65536 bind 65536) [ 0.912830] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.916338] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.919307] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.922592] NET: Registered protocol family 1 [ 0.925655] RPC: Registered named UNIX socket transport module. [ 0.928337] RPC: Registered udp transport module. [ 0.930457] RPC: Registered tcp transport module. [ 0.932427] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.935464] NET: Registered protocol family 44 [ 0.936977] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.938944] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.940652] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.942874] PCI: CLS 0 bytes, default 64 [ 0.944959] Unpacking initramfs... [ 2.311973] debug: unmapping init [mem 0xffff9368bcc54000-0xffff9368bffbffff] [ 2.315772] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.318377] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.321533] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.805658] Initialise system trusted keyrings [ 2.807263] Key type blacklist registered [ 2.809141] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.815241] zbud: loaded [ 2.817378] *** VALIDATE nfs *** [ 2.818538] *** VALIDATE nfs4 *** [ 2.820352] pstore: using deflate compression [ 2.823717] Platform Keyring initialized [ 2.930404] NET: Registered protocol family 38 [ 2.932712] Key type asymmetric registered [ 2.934438] Asymmetric key parser 'x509' registered [ 2.937161] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.941247] io scheduler mq-deadline registered [ 2.944092] io scheduler kyber registered [ 2.946218] io scheduler bfq registered [ 2.948144] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.952411] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.955818] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.958910] ACPI: Power Button [PWRF] [ 2.963776] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.969896] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.981788] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.988337] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.001829] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.031898] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.059520] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.064554] Non-volatile memory driver v1.3 [ 3.066463] Linux agpgart interface v0.103 [ 3.096716] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.099385] vda: detected capacity change from 0 to 74825728 [ 3.114840] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.118223] vdb: detected capacity change from 0 to 1073741824 [ 3.132862] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.135630] vdc: detected capacity change from 0 to 2621440000 [ 3.151077] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.154166] vdd: detected capacity change from 0 to 2621440000 [ 3.169253] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.172478] vde: detected capacity change from 0 to 4294967296 [ 3.187098] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.190631] vdf: detected capacity change from 0 to 4294967296 [ 3.200598] libphy: Fixed MDIO Bus: probed [ 3.205847] usbcore: registered new interface driver usbserial_generic [ 3.208288] usbserial: USB Serial support registered for generic [ 3.211050] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.216104] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.218223] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.220548] mousedev: PS/2 mouse device common for all mice [ 3.224360] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.224360] rtc_cmos 00:05: RTC can wake from S4 [ 3.226797] rtc_cmos 00:05: registered as rtc0 [ 3.234709] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.239442] intel_pstate: CPU model not supported [ 3.239987] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.247053] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.247063] hid: raw HID events driver (C) Jiri Kosina [ 3.253679] usbcore: registered new interface driver usbhid [ 3.257049] usbhid: USB HID core driver [ 3.259396] drop_monitor: Initializing network drop monitor service [ 3.263045] Initializing XFRM netlink socket [ 3.265978] NET: Registered protocol family 10 [ 3.270320] Segment Routing with IPv6 [ 3.272844] NET: Registered protocol family 17 [ 3.276643] mpls_gso: MPLS GSO support [ 3.282847] RAS: Correctable Errors collector initialized. [ 3.284881] AVX version of gcm_enc/dec engaged. [ 3.286946] AES CTR mode by8 optimization enabled [ 3.365897] sched_clock: Marking stable (3365872868, 0)->(4264132909, -898260041) [ 3.370128] registered taskstats version 1 [ 3.372750] Loading compiled-in X.509 certificates [ 3.375220] zswap: loaded using pool lzo/zbud [ 3.399847] Key type big_key registered [ 3.413363] Key type encrypted registered [ 3.415018] ima: No TPM chip found, activating TPM-bypass! [ 3.417584] ima: Allocated hash algorithm: sha1 [ 3.419540] ima: No architecture policies found [ 3.421564] evm: Initialising EVM extended attributes: [ 3.424609] evm: security.selinux [ 3.426010] evm: security.ima [ 3.427195] evm: security.capability [ 3.428569] evm: HMAC attrs: 0x1 [ 3.431412] rtc_cmos 00:05: setting system clock to 2026-08-23 18:16:54 UTC (1787509014) [ 3.439531] debug: unmapping init [mem 0xffffffff98603000-0xffffffff987fffff] [ 3.443325] debug: unmapping init [mem 0xffffffff97382000-0xffffffff97658fff] [ 3.452213] Write protecting the kernel read-only data: 28672k [ 3.456784] debug: unmapping init [mem 0xffffffff95a03000-0xffffffff95bfffff] [ 3.460696] debug: unmapping init [mem 0xffffffff96314000-0xffffffff963fffff] [ 3.495742] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.505323] systemd[1]: Detected virtualization kvm. [ 3.508265] systemd[1]: Detected architecture x86-64. [ 3.511327] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.538364] systemd[1]: No hostname configured. [ 3.540575] systemd[1]: Set hostname to . [ 3.542849] random: systemd: uninitialized urandom read (16 bytes read) [ 3.546327] systemd[1]: Initializing machine ID from random generator. [ 3.598410] random: ln: uninitialized urandom read (6 bytes read) [ 3.692631] random: systemd: uninitialized urandom read (16 bytes read) [ 3.695916] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.701723] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.706724] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.331122] device-mapper: uevent: version 1.0.3 [ 4.333807] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.122069] random: fast init done [ 5.159508] virtio_net virtio0 ens2: renamed from eth0 [ 5.295554] scsi host0: ata_piix [ 5.334154] scsi host1: ata_piix [ 5.336862] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.339588] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.527050] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.935486] random: crng init done [ 9.937186] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.343267] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.419104] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.660787] SELinux: Disabled at runtime. [ 11.719708] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.729308] systemd[1]: Detected virtualization kvm. [ 11.731311] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.154926] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.158336] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.162811] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.167055] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.171278] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.178729] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.183598] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ 12.222082] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.600056] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.827765] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.840208] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.958022] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.969061] EDAC sbridge: Ver: 1.1.2 [ 14.547806] Key type dns_resolver registered [ 14.863177] NFS: Registering the id_resolver key type [ 14.865431] Key type id_resolver registered [ 14.867289] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg344-server login: [ 40.509623] libcfs: loading out-of-tree module taints kernel. [ 40.533607] Key type ._llcrypt registered [ 40.535567] Key type .llcrypt registered [ 40.591431] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 49.464896] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 50.259229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.270814] alg: No test for adler32 (adler32-zlib) [ 51.324947] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 51.693858] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 53.335197] Key type lgssc registered [ 54.107334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 69.889860] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 116.249144] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 131.828119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 131.870625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 133.292031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 133.332295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 133.468459] Lustre: lustre-MDT0000: new disk, initializing [ 133.629403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 133.668342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 135.820042] hrtimer: interrupt took 6947619 ns [ 139.517916] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 155.590805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.709047] Lustre: 6489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 155.738128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 155.746163] Lustre: Skipped 1 previous similar message [ 155.858324] Lustre: lustre-MDT0001: new disk, initializing [ 155.972193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 156.012296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 156.032227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 160.870303] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.643516] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 178.584440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 178.979596] Lustre: lustre-OST0000: new disk, initializing [ 178.992862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 179.004816] Lustre: 8426:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 179.119625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.905552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 185.913859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 186.048166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 187.351154] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.592580] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.986613] Lustre: lustre-OST0001: new disk, initializing [ 207.996226] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 208.012451] Lustre: 9499:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 208.186772] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.136042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 216.158365] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 216.165461] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.238911] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 229.842625] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 238.456893] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 244.890811] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing check_logdir /tmp/testlogs/ [ 250.683964] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing yml_node [ 255.896878] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 259.181882] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 262.276923] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 264.096574] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 23 14:21:14 EDT 2026 [ 283.531559] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 285.580395] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 303.585223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.592628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.611933] Lustre: Skipped 3 previous similar messages [ 303.634386] Lustre: Skipped 3 previous similar messages [ 307.971614] Lustre: server umount lustre-MDT0000 complete [ 312.392607] LustreError: 6481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787509323 with bad export cookie 10486666124994755914 [ 312.402232] LustreError: 6481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 312.404681] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.823890] LustreError: 9458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.823987] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.825424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.836037] LustreError: 9458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 313.872812] Lustre: Skipped 1 previous similar message [ 317.855939] LustreError: 6497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.856305] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 317.873050] LustreError: 6497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 317.895101] Lustre: Skipped 2 previous similar messages [ 318.803252] Lustre: server umount lustre-MDT0001 complete [ 323.025507] Lustre: server umount lustre-OST0000 complete [ 326.970963] Lustre: server umount lustre-OST0001 complete [ 334.314483] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 343.560475] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 379.784281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.035540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 380.061313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 380.159489] Lustre: lustre-MDT0000: new disk, initializing [ 380.247453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 380.270953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 385.487474] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 396.634445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.786198] Lustre: 15056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 396.851068] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 396.857529] Lustre: Skipped 1 previous similar message [ 396.968287] Lustre: lustre-MDT0001: new disk, initializing [ 397.133360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 397.188332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 397.203980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 402.050231] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 411.890363] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 418.673946] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 426.546800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 427.026779] Lustre: lustre-OST0000: new disk, initializing [ 427.033994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 427.044973] Lustre: 17028:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 427.185883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.383790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 428.394241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 428.540625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 434.194730] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 445.551857] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 453.021667] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 453.448669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 461.476628] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 461.798101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 464.352151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 464.356916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.370619] LustreError: Skipped 1 previous similar message [ 464.397684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 464.418776] Lustre: Skipped 1 previous similar message [ 469.478654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 469.501142] Lustre: Skipped 1 previous similar message [ 470.468508] Lustre: server umount lustre-OST0000 complete [ 479.208076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 479.214151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.233311] Lustre: Skipped 1 previous similar message [ 479.243179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 481.418279] Lustre: server umount lustre-MDT0000 complete [ 484.833308] LustreError: 15935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 484.855348] LustreError: 15935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 485.201928] LustreError: 15048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787509496 with bad export cookie 10486666124994757447 [ 485.213252] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 485.214211] LustreError: 15048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 485.664646] Lustre: server umount lustre-MDT0001 complete [ 491.428490] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:25:01 (1787509501) [ 511.078979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.733077] LustreError: 19904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.836330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.547612] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 517.096320] LustreError: 19905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.450691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.819467] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 539.497892] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 545.923234] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 548.077230] Lustre: Failing over lustre-MDT0000 [ 548.564337] Lustre: server umount lustre-MDT0000 complete [ 550.188799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.205263] Lustre: Skipped 1 previous similar message [ 550.221040] LustreError: 19900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.249512] LustreError: 19900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 566.623830] Lustre: 3621:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787509561/real 1787509561] req@ffff936803714700 x1874339102363776/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787509577 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 566.651459] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 566.676251] LustreError: 20912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.698196] LustreError: 20912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 570.398176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.930381] LustreError: 3620:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff936904e83800 x1874339102366720/t0(0) o250->MGC192.168.203.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 577.410422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.419883] Lustre: Skipped 1 previous similar message [ 582.631675] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 582.643259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 582.698133] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 583.719992] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 595.221756] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 600.032456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.038127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 600.069828] Lustre: Skipped 2 previous similar messages [ 600.095045] Lustre: Skipped 2 previous similar messages [ 614.368487] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 614.580369] Lustre: server umount lustre-MDT0000 complete [ 615.400633] LustreError: 20912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 615.421794] LustreError: 20912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 618.568741] LustreError: 19884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787509629 with bad export cookie 10486666124994758861 [ 618.581536] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.590787] LustreError: 19884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 619.033800] Lustre: server umount lustre-MDT0001 complete [ 627.889385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.084975] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 641.750979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.814586] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 654.280284] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 659.441727] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 661.316802] Lustre: Failing over lustre-MDT0000 [ 661.483292] Lustre: server umount lustre-MDT0000 complete [ 663.014382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 663.027361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 680.327779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.532967] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 680.849981] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 685.240927] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 686.054249] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 686.060789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 686.071668] Lustre: Skipped 1 previous similar message [ 686.098338] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 694.216763] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 698.327685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 698.343649] Lustre: Skipped 1 previous similar message [ 698.350898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 698.359499] Lustre: Skipped 5 previous similar messages [ 704.714058] Lustre: server umount lustre-MDT0000 complete [ 708.578420] LustreError: 23243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 708.589605] LustreError: 23243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 708.843182] LustreError: 23224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787509719 with bad export cookie 10486666124994759799 [ 708.845392] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 708.853626] LustreError: 23224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 709.277615] Lustre: server umount lustre-MDT0001 complete [ 718.151569] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:28:48 (1787509728) [ 733.862595] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 743.903927] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 782.270309] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 792.881973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 793.132980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 793.179268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 793.264781] Lustre: lustre-MDT0000: new disk, initializing [ 793.414114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 793.418646] Lustre: Skipped 3 previous similar messages [ 793.443728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 798.150187] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 810.035780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.257091] Lustre: 29614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 810.555284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 815.200622] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 825.254218] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 831.262661] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 837.916104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 838.261212] Lustre: 31582:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 839.893774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 839.947428] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 840.039799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 840.057357] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 844.534233] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 853.877170] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 859.408955] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 881.122081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 881.127044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 881.130156] LustreError: Skipped 1 previous similar message [ 881.130888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 881.130891] Lustre: Skipped 2 previous similar messages [ 881.173894] Lustre: Skipped 2 previous similar messages [ 882.911641] Lustre: server umount lustre-OST0000 complete [ 887.264882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 901.087224] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 901.309076] Lustre: server umount lustre-MDT0000 complete [ 901.600773] LustreError: 29622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 901.611261] LustreError: 29622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 904.758082] LustreError: 30409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787509915 with bad export cookie 10486666124994760205 [ 904.759898] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 904.777643] LustreError: 30409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 905.357144] Lustre: server umount lustre-MDT0001 complete [ 915.847752] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 918.443766] Key type lgssc unregistered [ 918.741624] LNet: 33563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.753520] LNetError: 33563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.788871] LNet: Removed LNI 192.168.203.144@tcp [ 919.817187] Key type .llcrypt unregistered [ 919.819837] Key type ._llcrypt unregistered [ 942.718562] Key type ._llcrypt registered [ 942.722186] Key type .llcrypt registered [ 942.825746] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 943.639775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 943.653429] alg: No test for adler32 (adler32-zlib) [ 944.710247] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 944.936145] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 946.633054] Key type lgssc registered [ 947.816542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 954.696520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 954.708257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 956.134832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.204932] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 966.628396] LustreError: 34978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 966.652607] LustreError: 34978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 967.882565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.230662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 972.317701] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 978.564318] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 984.093464] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 991.097880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 991.448790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.540093] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 996.544224] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1001.752690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1003.595879] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1017.361366] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:33:47 (1787510027) [ 1019.246958] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1020.910511] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1027.714607] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:33:57 (1787510037) [ 1035.754293] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:34:05 (1787510045) [ 1036.739344] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1045.528160] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1045.532621] Lustre: Skipped 2 previous similar messages [ 1047.742305] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:34:17 (1787510057) [ 1049.867873] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1049.874065] Lustre: Skipped 2 previous similar messages [ 1058.821330] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1058.834630] Lustre: Skipped 4 previous similar messages [ 1073.622779] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:34:43 (1787510083) [ 1076.786916] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1080.367124] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1089.007865] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:34:58 (1787510098) [ 1091.142929] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1091.147302] Lustre: Skipped 6 previous similar messages [ 1103.220970] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:35:13 (1787510113) [ 1116.643763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1116.649845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.657472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1119.713748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1119.718054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.721755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1122.256812] Lustre: server umount lustre-OST0000 complete [ 1126.880451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1126.887297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.921316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1126.930697] Lustre: Skipped 1 previous similar message [ 1129.957486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.959817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.089529] Lustre: server umount lustre-MDT0000 complete [ 1135.091837] LustreError: 34974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.110270] LustreError: 34974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1135.809486] LustreError: 38542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787510146 with bad export cookie 8155651228192956183 [ 1135.810781] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.827182] LustreError: 38542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1136.209684] Lustre: server umount lustre-MDT0001 complete [ 1146.815629] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1149.683936] Key type lgssc unregistered [ 1150.015880] LNet: 42844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.026830] LNetError: 42844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1150.040343] LNet: Removed LNI 192.168.203.144@tcp [ 1150.974155] Key type .llcrypt unregistered [ 1150.978194] Key type ._llcrypt unregistered [ 1168.112151] Key type ._llcrypt registered [ 1168.113869] Key type .llcrypt registered [ 1168.226193] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 1183.968834] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1185.001822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1185.165292] alg: No test for adler32 (adler32-zlib) [ 1186.207196] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 1186.498326] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1188.199221] Key type lgssc registered [ 1189.305928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1224.756142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1224.779716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.074684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1226.101942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1226.184896] Lustre: lustre-MDT0000: new disk, initializing [ 1226.259247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.272961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1231.134502] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1241.539712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.653660] Lustre: 45996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1241.708511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1241.713239] Lustre: Skipped 1 previous similar message [ 1241.811420] Lustre: lustre-MDT0001: new disk, initializing [ 1241.914866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1241.950597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1241.966250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1246.444661] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1255.243872] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1261.723905] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1268.210212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1268.575259] Lustre: lustre-OST0000: new disk, initializing [ 1268.589794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1268.599338] Lustre: 47967:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1268.711361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1274.426729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1274.450931] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1274.601810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1275.188106] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1284.222473] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1289.754222] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1289.958556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1294.698703] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1295.035490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1300.450736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1300.459647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.463780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1300.484904] Lustre: Skipped 1 previous similar message [ 1303.428089] Lustre: server umount lustre-OST0000 complete [ 1308.642576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1308.651430] LustreError: Skipped 1 previous similar message [ 1308.654893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.685546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1308.694484] Lustre: Skipped 1 previous similar message [ 1310.688523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1310.691170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.909146] Lustre: server umount lustre-MDT0000 complete [ 1317.548378] LustreError: 45990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787510328 with bad export cookie 9686127087599106482 [ 1317.549421] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1317.565028] LustreError: 45990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1318.001246] Lustre: server umount lustre-MDT0001 complete [ 1327.594250] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:38:57 (1787510337) [ 1333.948320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.423978] LustreError: 50428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1334.450742] LustreError: 50428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1334.511729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1339.884766] LustreError: 50429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.311344] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1343.971202] LustreError: 50428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.100452] LustreError: 50429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.971940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.210461] LustreError: 50428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.724567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1360.856388] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1370.101548] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1377.284138] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1386.042650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1386.648304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.083175] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1402.204565] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1411.265527] LustreError: 52961:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1412.330916] Lustre: Modifying parameter general.timeout=1 in log params [ 1413.381775] Lustre: Modifying parameter general.timeout=2 in log params [ 1416.263591] Lustre: Modifying parameter general.timeout=5 in log params [ 1416.272280] Lustre: Skipped 2 previous similar messages [ 1420.638597] Lustre: Modifying parameter general.timeout=9 in log params [ 1420.645024] Lustre: Skipped 3 previous similar messages [ 1429.164050] Lustre: Modifying parameter general.timeout=17 in log params [ 1429.167751] Lustre: Skipped 7 previous similar messages [ 1445.533137] Lustre: Modifying parameter general.timeout=35 in log params [ 1445.538773] Lustre: Skipped 17 previous similar messages [ 1478.151826] Lustre: Modifying parameter general.timeout=75 in log params [ 1478.158701] Lustre: Skipped 39 previous similar messages [ 1538.036821] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:42:27 (1787510547) [ 1540.906876] Lustre: 59904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1542.764099] Lustre: 60003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1543.816576] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1543.828789] Lustre: Skipped 58 previous similar messages [ 1544.883088] Lustre: 60101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1550.394208] Lustre: 60413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1550.408697] Lustre: 60413:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1577.603686] Lustre: 61775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1577.630301] LustreError: 61775:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1581.733224] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:43:11 (1787510591) [ 1595.365638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1595.384601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.406366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1595.411838] Lustre: Skipped 1 previous similar message [ 1600.995793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1601.001160] Lustre: Skipped 4 previous similar messages [ 1601.351471] Lustre: server umount lustre-MDT0000 complete [ 1604.590541] LustreError: 51164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787510615 with bad export cookie 9686127087599107364 [ 1604.593239] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1604.598673] LustreError: 51164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1604.870852] Lustre: server umount lustre-MDT0001 complete [ 1618.411067] Lustre: server umount lustre-OST0000 complete [ 1646.276384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.562956] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1646.938511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1651.493977] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1660.892946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.071970] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1661.100863] Lustre: 64310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1667.640522] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1677.728630] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1685.147950] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1693.260385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1693.448838] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1693.692827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1693.698838] Lustre: Skipped 1 previous similar message [ 1700.140274] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1706.866155] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1708.524197] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1708.544539] Lustre: Skipped 19 previous similar messages [ 1708.605368] Lustre: 66877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1716.704879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1716.709708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1716.726548] Lustre: Skipped 2 previous similar messages [ 1716.741364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1721.435578] Lustre: server umount lustre-OST0000 complete [ 1727.457445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1727.470940] Lustre: Skipped 1 previous similar message [ 1731.330930] Lustre: server umount lustre-MDT0000 complete [ 1732.584449] LustreError: 64320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1732.602176] LustreError: 64320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1736.026845] LustreError: 64300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787510747 with bad export cookie 9686127087599110381 [ 1736.032326] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1736.032593] LustreError: 64300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1736.545585] Lustre: server umount lustre-MDT0001 complete [ 1751.171230] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1754.055299] Key type lgssc unregistered [ 1754.433712] LNet: 68089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.453368] LNetError: 68089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.490884] LNet: Removed LNI 192.168.203.144@tcp [ 1755.566221] Key type .llcrypt unregistered [ 1755.570544] Key type ._llcrypt unregistered [ 1780.161233] Key type ._llcrypt registered [ 1780.163940] Key type .llcrypt registered [ 1780.283611] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1781.432868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1781.594728] alg: No test for adler32 (adler32-zlib) [ 1782.739915] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 1783.060572] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1784.783193] Key type lgssc registered [ 1786.236259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1794.239587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1794.262447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.735416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.206821] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1805.286863] LustreError: 69502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1805.308905] LustreError: 69502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1808.123627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1808.903651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1813.796602] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1821.910406] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1828.407521] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1835.395836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1836.087462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1842.615814] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1850.174661] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1861.211481] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1870.309495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1870.318148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.348723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1873.889270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1873.889896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.906234] Lustre: Skipped 1 previous similar message [ 1873.926210] Lustre: Skipped 1 previous similar message [ 1876.032850] Lustre: server umount lustre-MDT0000 complete [ 1879.013491] LustreError: 69498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1879.030092] LustreError: 69498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1879.577158] LustreError: 69484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787510890 with bad export cookie 6571534109470136298 [ 1879.589581] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1879.592880] LustreError: 69484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1879.909597] Lustre: server umount lustre-MDT0001 complete [ 1894.258249] Lustre: server umount lustre-OST0000 complete [ 1920.018964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1920.383869] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1920.435587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1920.750972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.898871] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1933.933927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1934.068163] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1934.117453] Lustre: 74373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1934.236495] LustreError: 74385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1934.758873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1939.443551] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1947.890769] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1953.763562] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1960.805727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1961.012517] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1961.571530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1967.459435] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1974.566775] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1976.238260] Lustre: 76948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1978.220334] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1987.047684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1987.055618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.071587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1987.084340] Lustre: Skipped 1 previous similar message [ 1989.357971] Lustre: server umount lustre-OST0000 complete [ 1995.744099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1995.748861] LustreError: Skipped 1 previous similar message [ 1995.752150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.766838] Lustre: Skipped 1 previous similar message [ 1995.780980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1995.790746] Lustre: Skipped 1 previous similar message [ 1999.253949] Lustre: server umount lustre-MDT0000 complete [ 2002.591901] LustreError: 75136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787511013 with bad export cookie 6571534109470137138 [ 2002.601675] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2002.606638] LustreError: 75136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2002.981824] Lustre: server umount lustre-MDT0001 complete [ 2013.203171] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2015.686513] Key type lgssc unregistered [ 2016.028477] LNet: 78165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.035498] LNetError: 78165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2017.061745] LNet: Removed LNI 192.168.203.144@tcp [ 2017.998148] Key type .llcrypt unregistered [ 2018.004688] Key type ._llcrypt unregistered [ 2041.683268] Key type ._llcrypt registered [ 2041.686267] Key type .llcrypt registered [ 2041.803037] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2042.830808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2042.882474] alg: No test for adler32 (adler32-zlib) [ 2043.972374] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 2044.242843] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2045.895186] Key type lgssc registered [ 2046.782473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2054.161705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2054.175867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2055.542241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.780214] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2065.891715] LustreError: 79578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.914647] LustreError: 79578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2067.706925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.325933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2072.461228] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2079.545251] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2085.055372] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2091.246437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2091.666667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.387613] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2103.697452] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2110.691580] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2111.586656] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2113.343830] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2113.347286] Lustre: Skipped 1 previous similar message [ 2115.830212] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2115.833983] Lustre: Skipped 2 previous similar messages [ 2120.331482] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2120.334622] Lustre: Skipped 5 previous similar messages [ 2128.626862] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2128.630324] Lustre: Skipped 9 previous similar messages [ 2145.010323] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2145.019050] Lustre: Skipped 20 previous similar messages [ 2177.072869] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2177.076851] Lustre: Skipped 40 previous similar messages [ 2196.924155] Lustre: 87670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2199.252665] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 14:53:29 (1787511209) [ 2200.576756] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2202.200391] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 14:53:32 (1787511212) [ 2206.690851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2206.698315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.721321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2208.737014] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2208.740131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2208.752275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.277822] Lustre: server umount lustre-OST0000 complete [ 2216.417693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2216.427716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.446173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2216.450982] Lustre: Skipped 1 previous similar message [ 2218.978540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2218.979807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.881066] Lustre: server umount lustre-MDT0000 complete [ 2224.096557] LustreError: 79574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.111120] LustreError: 79574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2225.121759] LustreError: 79559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787511236 with bad export cookie 9960960194402882927 [ 2225.122895] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2225.133763] LustreError: 79559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2225.421294] Lustre: server umount lustre-MDT0001 complete [ 2235.438784] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2238.118753] Key type lgssc unregistered [ 2238.426657] LNet: 88921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.434150] LNetError: 88921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.457408] LNet: Removed LNI 192.168.203.144@tcp [ 2239.346342] Key type .llcrypt unregistered [ 2239.350371] Key type ._llcrypt unregistered [ 2243.169692] Key type ._llcrypt registered [ 2243.175996] Key type .llcrypt registered [ 2243.286300] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2247.444495] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules [ 2248.545174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2248.563307] alg: No test for adler32 (adler32-zlib) [ 2249.584960] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 2249.795980] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2251.535171] Key type lgssc registered [ 2252.803819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2253.949405] LustreError: 90124:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2269.916187] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2275.480568] LustreError: 90124:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2277.533598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2277.562521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2279.008052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.747421] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2288.099457] LustreError: 91149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.119523] LustreError: 91149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2289.710876] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 14:55:00 (1787511300) [ 2293.189238] LustreError: 91180:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2293.208691] LustreError: 91180:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2293.223638] LustreError: 91180:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 2293.507621] Lustre: server umount lustre-MDT0000 complete [ 2303.906722] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2306.165802] Key type lgssc unregistered [ 2306.434550] LNet: 92676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.442038] LNetError: 92676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2306.458146] LNet: Removed LNI 192.168.203.144@tcp [ 2307.197994] Key type .llcrypt unregistered [ 2307.200718] Key type ._llcrypt unregistered [ 2321.216790] Key type ._llcrypt registered [ 2321.219399] Key type .llcrypt registered [ 2321.294284] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2322.065193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2322.080985] alg: No test for adler32 (adler32-zlib) [ 2323.067562] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 2323.261734] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2324.899110] Key type lgssc registered [ 2325.796853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2332.276202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2332.299123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2333.762863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.830708] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2343.909088] LustreError: 93872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.939289] LustreError: 93872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2345.155611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.762337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2349.961170] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2356.675517] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2362.220586] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2367.935148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2368.445841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2373.249435] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2379.155398] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2420.848193] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 14:57:11 (1787511431) [ 2422.208569] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2423.674918] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 14:57:13 (1787511433) [ 2427.364758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2427.375775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2427.388618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.920164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.922096] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.927217] Lustre: Skipped 1 previous similar message [ 2429.945373] Lustre: Skipped 1 previous similar message [ 2432.447490] Lustre: server umount lustre-MDT0000 complete [ 2435.052353] LustreError: 94925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.071799] LustreError: 94925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2435.985964] LustreError: 97029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787511447 with bad export cookie 14676921997335332244 [ 2435.990270] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2435.998045] LustreError: 97029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2436.292247] Lustre: server umount lustre-MDT0001 complete [ 2450.176883] Lustre: server umount lustre-OST0000 complete [ 2459.997857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.556611] LustreError: 98146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.776801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2465.324524] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2466.272918] LustreError: 98147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.381230] LustreError: 98146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2473.418600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.136757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2478.500445] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2485.384802] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2489.979745] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2502.142333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2502.343776] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2502.351759] LustreError: 98136:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2502.371798] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2502.377840] LustreError: 100229:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2502.385225] LustreError: 100229:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2502.390361] LustreError: 100229:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2502.463425] Lustre: server umount lustre-OST0000 complete [ 2502.472621] LustreError: 100229:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2510.441705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2510.697712] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2510.702944] LustreError: 98134:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2510.717586] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2510.725188] LustreError: 100692:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2510.730204] LustreError: 100692:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2510.737927] LustreError: 100692:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2510.796272] Lustre: server umount lustre-OST0000 complete [ 2510.804870] LustreError: 100692:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2519.565185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2519.737414] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2519.743660] Lustre: Found index 0 for lustre-OST0000, updating log [ 2519.747194] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2519.808927] Lustre: lustre-OST0000: new disk, initializing [ 2519.812551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2519.817803] Lustre: 101227:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2520.040967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2525.642518] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2529.459553] Lustre: Failing over lustre-OST0000 [ 2529.662603] Lustre: server umount lustre-OST0000 complete [ 2535.400136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2535.407483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.425829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2539.789693] Lustre: server umount lustre-MDT0000 complete [ 2541.030774] LustreError: 99194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2541.062241] LustreError: 99194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2543.472545] LustreError: 101226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787511554 with bad export cookie 14676921997335334988 [ 2543.475487] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2543.488699] LustreError: 101226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2543.846594] Lustre: server umount lustre-MDT0001 complete [ 2551.521719] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 14:59:21 (1787511561) [ 2566.448748] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2576.238081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.597268] LustreError: 103586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.627552] LustreError: 103586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2576.756542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2580.418747] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2588.184204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.647804] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2596.355446] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2602.619139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2608.212387] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2615.964835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2616.085484] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2616.147940] Lustre: lustre-OST0001: new disk, initializing [ 2616.154900] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2616.167453] Lustre: 105904:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2616.463480] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2616.472124] Lustre: Skipped 2 previous similar messages [ 2621.882271] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2622.534821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2622.549290] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2622.667479] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2628.633319] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2634.054642] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2638.310839] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2638.313965] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2638.342351] Lustre: Skipped 3 previous similar messages [ 2638.351257] Lustre: Skipped 2 previous similar messages [ 2643.428968] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2643.439873] Lustre: Skipped 3 previous similar messages [ 2652.127317] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2652.367144] Lustre: server umount lustre-MDT0001 complete [ 2653.667210] LustreError: 104317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.678810] LustreError: 104317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2661.366185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.486656] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2661.492167] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2661.537927] Lustre: 103576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2663.109395] LustreError: 107489:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2663.121219] LustreError: 107489:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2663.129598] LustreError: 103579:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2663.260540] LustreError: 107501:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 2663.271586] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2663.271586] [ 2666.162479] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2667.497205] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2675.612455] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 15:01:25 (1787511685) [ 2692.223802] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2708.129020] Lustre: 109532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2728.651771] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2732.049451] Lustre: 110667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2815.456967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2815.470737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.495420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.501893] Lustre: Skipped 5 previous similar messages [ 2818.004789] Lustre: server umount lustre-MDT0000 complete [ 2821.090897] LustreError: 107756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.103968] LustreError: 107756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2821.324294] LustreError: 107409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787511832 with bad export cookie 14676921997335335905 [ 2821.335679] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2821.337976] LustreError: 107409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2821.616892] Lustre: server umount lustre-MDT0001 complete [ 2835.526484] Lustre: server umount lustre-OST0000 complete [ 2849.017980] Lustre: server umount lustre-OST0001 complete [ 2854.245358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2862.834432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2869.967051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2878.390189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2895.125547] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2903.486350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2903.507818] Lustre: lustre-MDT0000: reset Object Index mappings [ 2903.916184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2903.925135] Lustre: Skipped 1 previous similar message [ 2907.632231] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2914.575472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2914.596711] Lustre: lustre-MDT0001: reset Object Index mappings [ 2919.151539] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2921.910330] Lustre: 115766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2928.152908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2934.284467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2934.929836] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2940.403597] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 2943.148449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2949.105312] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2949.128789] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 2949.530387] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2956.876703] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2960.144094] Lustre: 117644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3000.871470] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 15:06:51 (1787512011) [ 3031.015762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.018840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3031.035314] Lustre: Skipped 6 previous similar messages [ 3031.046310] Lustre: Skipped 6 previous similar messages [ 3042.271734] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3042.467988] Lustre: server umount lustre-MDT0000 complete [ 3045.643837] LustreError: 115368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787512056 with bad export cookie 14676921997335382784 [ 3045.644745] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3045.660126] LustreError: 115368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3046.369145] LustreError: 116433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.387225] LustreError: 116433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 3052.048837] Lustre: server umount lustre-MDT0001 complete [ 3061.317944] Lustre: server umount lustre-OST0000 complete [ 3064.103319] LustreError: 118602:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000401:0x7d:0x0].0x0 (ffff93680cadb200) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3070.399870] Lustre: server umount lustre-OST0001 complete [ 3076.118993] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 3084.671327] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3112.860473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3113.050225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3113.071368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3113.147192] Lustre: lustre-MDT0000: new disk, initializing [ 3113.220566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.224705] Lustre: Skipped 3 previous similar messages [ 3113.240398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3116.550050] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3122.436669] Lustre: server umount lustre-MDT0000 complete [ 3129.271129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3129.491877] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3129.505783] Lustre: 121911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3129.518900] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3133.080440] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3137.063381] Lustre: server umount lustre-MDT0000 complete [ 3144.368724] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 15:09:14 (1787512154) [ 3145.559512] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3146.958963] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 15:09:17 (1787512157) [ 3160.508850] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3168.940277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3172.646923] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3179.319379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3179.436273] Lustre: 123959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3179.484530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3179.490019] Lustre: Skipped 1 previous similar message [ 3179.568767] Lustre: lustre-MDT0001: new disk, initializing [ 3179.711933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3179.719591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3183.362809] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3187.344318] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3192.823269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3193.104689] Lustre: lustre-OST0000: new disk, initializing [ 3193.111894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3193.128050] Lustre: 125536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3194.866975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3194.882519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3195.017214] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3198.247833] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3207.988923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3208.139502] Lustre: lustre-OST0001: new disk, initializing [ 3208.143161] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3208.152400] Lustre: 126407:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3209.420915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3209.433474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3209.535828] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3212.956778] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3220.978665] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3223.948385] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3515.402892] Lustre: 132803:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3515.409683] Lustre: 132803:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3515.415212] Lustre: 132803:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3515.421347] Lustre: 132803:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3515.427260] Lustre: 132803:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3515.434400] Lustre: 132803:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3678.198549] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:18:08 (1787512688) [ 3690.982605] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3690.985194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3690.994403] Lustre: Skipped 5 previous similar messages [ 3691.001110] Lustre: Skipped 15 previous similar messages [ 3696.102961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3696.109795] Lustre: Skipped 2 previous similar messages [ 3705.311147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3705.455349] Lustre: server umount lustre-MDT0000 complete [ 3706.338242] LustreError: 123971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3706.350100] LustreError: 123971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3707.321295] LustreError: 124707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787512718 with bad export cookie 14676921997335448885 [ 3707.326040] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3707.326544] LustreError: 124707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3711.458268] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.462350] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3711.466920] Lustre: Skipped 1 previous similar message [ 3711.471538] Lustre: Skipped 4 previous similar messages [ 3713.632519] Lustre: server umount lustre-MDT0001 complete [ 3729.887814] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3729.935618] Lustre: server umount lustre-OST0000 complete [ 3746.271422] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3746.387244] Lustre: server umount lustre-OST0001 complete [ 3749.682874] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 3755.078510] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3774.150981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3774.297937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3774.325866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3774.395959] Lustre: lustre-MDT0000: new disk, initializing [ 3774.448802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3774.452534] Lustre: Skipped 5 previous similar messages [ 3774.481388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3776.782870] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3783.056027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.128843] Lustre: 135841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3783.318423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3785.568418] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3790.677736] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3793.380821] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3796.915384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3797.061664] Lustre: 137812:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3798.989203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3798.995561] Lustre: Skipped 1 previous similar message [ 3799.000060] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3799.058297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3800.101215] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3805.647483] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3810.653275] Lustre: lustre-MDD0000: changelog on [ 3812.118169] Lustre: lustre-MDD0001: changelog on [ 4683.588988] Lustre: lustre-MDD0001: changelog off [ 4684.475253] Lustre: 136291:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4684.941765] Lustre: lustre-MDD0000: changelog off [ 4689.888970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4689.890231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.891878] LustreError: Skipped 1 previous similar message [ 4689.897632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4689.901577] Lustre: Skipped 4 previous similar messages [ 4693.360728] Lustre: server umount lustre-OST0000 complete [ 4695.008872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.009119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4695.014743] Lustre: Skipped 1 previous similar message [ 4695.021547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4695.024416] Lustre: Skipped 1 previous similar message [ 4700.128304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4700.131444] Lustre: Skipped 1 previous similar message [ 4709.343115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4709.449995] Lustre: server umount lustre-MDT0000 complete [ 4710.368190] LustreError: 136651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.375701] LustreError: 136651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4710.834059] LustreError: 135835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787513721 with bad export cookie 14676921997336217926 [ 4710.836048] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4710.839938] LustreError: 135835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4710.978807] Lustre: server umount lustre-MDT0001 complete [ 4715.204764] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 4716.276888] Key type lgssc unregistered [ 4716.411463] LNet: 146727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4716.414807] LNetError: 146727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4716.422333] LNet: Removed LNI 192.168.203.144@tcp [ 4716.756155] Key type .llcrypt unregistered [ 4716.758193] Key type ._llcrypt unregistered [ 4723.974393] Key type ._llcrypt registered [ 4723.976177] Key type .llcrypt registered [ 4724.019244] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 4730.771895] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 4731.172384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4731.202719] alg: No test for adler32 (adler32-zlib) [ 4732.059652] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 4732.154651] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 4733.743143] Key type lgssc registered [ 4734.193114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4748.383119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4748.391693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4749.498510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4749.513926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4749.557268] Lustre: lustre-MDT0000: new disk, initializing [ 4749.586570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4749.596163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4751.388751] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4756.744697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4756.796504] Lustre: 149901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4756.831915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4756.834618] Lustre: Skipped 1 previous similar message [ 4756.880694] Lustre: lustre-MDT0001: new disk, initializing [ 4756.926112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4756.941465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4756.946200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4758.619866] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4762.910896] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4765.179671] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4768.014519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4768.162829] Lustre: lustre-OST0000: new disk, initializing [ 4768.165861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4768.169965] Lustre: 151869:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4768.204140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4770.632672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4770.637346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4770.694544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4770.783747] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4775.434779] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4777.954398] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_module obdecho/obdecho [ 4778.706537] LustreError: 152872:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4778.710276] LustreError: 152872:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 4779.448917] LustreError: 152968:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 4786.146726] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4786.148812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4786.151465] Lustre: Skipped 2 previous similar messages [ 4786.156186] Lustre: Skipped 2 previous similar messages [ 4789.480659] Lustre: server umount lustre-MDT0000 complete [ 4790.961171] LustreError: 149892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787513802 with bad export cookie 18315477882679491504 [ 4790.966379] LustreError: 149892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4790.966576] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4791.101911] Lustre: server umount lustre-MDT0001 complete [ 4802.927908] Lustre: server umount lustre-OST0000 complete [ 4811.876756] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 4816.231420] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 4831.346621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4831.462714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4831.478936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4831.526839] Lustre: lustre-MDT0000: new disk, initializing [ 4831.565195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4831.574530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4833.208814] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4838.012932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4838.063176] Lustre: 156676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4838.092235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4838.095262] Lustre: Skipped 1 previous similar message [ 4838.130923] Lustre: lustre-MDT0001: new disk, initializing [ 4838.173585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4838.189861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4838.194057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4839.853550] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4843.932057] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4846.056936] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4848.659068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4848.779287] Lustre: lustre-OST0000: new disk, initializing [ 4848.781795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4848.788244] Lustre: 158646:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4848.823605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4850.745812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4850.750846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4850.786684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4850.999604] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4854.972134] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4856.561458] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4856.650887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4858.228290] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4858.305882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4860.897420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4860.898075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4860.907695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4860.910398] Lustre: Skipped 1 previous similar message [ 4865.173199] Lustre: server umount lustre-OST0000 complete [ 4869.087923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4869.091246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4869.097097] Lustre: Skipped 1 previous similar message [ 4869.099825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4871.136141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4871.136654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4872.605829] Lustre: server umount lustre-MDT0000 complete [ 4873.857956] LustreError: 156667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787513884 with bad export cookie 18315477882679492470 [ 4873.861599] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4873.862399] LustreError: 156667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4873.978917] Lustre: server umount lustre-MDT0001 complete [ 4875.209732] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:38:06 (1787513886) [ 4875.766592] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4876.398936] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 15:38:07 (1787513887) [ 4879.249784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4879.455324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4880.960317] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4884.265555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4885.968952] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4888.619284] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4890.525133] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4892.840930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4894.949430] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4897.432287] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4910.408280] LustreError: 163654:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd1:0x2f:0x0].0x0 (000000006732b905) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4913.328908] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 4913.333795] Lustre: Skipped 1 previous similar message [ 4914.657274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4914.662423] Lustre: Skipped 2 previous similar messages [ 4916.577808] Lustre: server umount lustre-MDT0000 complete [ 4917.910976] LustreError: 161036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787513928 with bad export cookie 18315477882679493352 [ 4917.913662] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4917.915732] LustreError: 161036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4918.436174] LustreError: 162435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4918.444796] LustreError: 162435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4919.775801] LustreError: 162435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.783787] LustreError: 161050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.784572] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4922.794151] Lustre: Skipped 5 previous similar messages [ 4924.079569] Lustre: server umount lustre-MDT0001 complete [ 4926.410613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4931.186658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4931.280084] LustreError: 164914:0:(llog.c:1655:llog_backup()) MGC192.168.203.144@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4931.283881] Lustre: 164914:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4947.423906] LustreError: 164918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4947.428986] LustreError: 164918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4952.543854] LustreError: 164937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4952.550148] LustreError: 164937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4962.783965] LustreError: 164918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4962.791593] LustreError: 164918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4978.852414] LustreError: 164920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4978.860185] LustreError: 164920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 4992.479359] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4992.481516] Lustre: 164914:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5011.936447] LustreError: 164919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5011.942323] LustreError: 164919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 5032.934183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5032.937691] Lustre: Skipped 2 previous similar messages [ 5034.373480] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5037.029105] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5037.819392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5062.629786] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfe2da61840a29c33 [ 5062.635851] Lustre: MGC192.168.203.144@tcp: Connection restored to 0@lo (at 0@lo) [ 5064.179732] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5066.679521] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5067.232518] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5067.749618] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5069.063424] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5069.597306] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5076.694457] Lustre: server umount lustre-MDT0000 complete [ 5077.879378] LustreError: 161035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787514088 with bad export cookie 18315477882679499827 [ 5077.882977] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5077.883443] LustreError: 161035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5077.995723] Lustre: server umount lustre-MDT0001 complete [ 5099.667990] Lustre: server umount lustre-OST0000 complete [ 5106.161087] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 5109.271069] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5121.604409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5121.687386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5121.696961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5121.732209] Lustre: lustre-MDT0000: new disk, initializing [ 5121.750113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5121.752891] Lustre: Skipped 1 previous similar message [ 5121.756943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5123.037040] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5127.203930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5127.242419] Lustre: 169992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5127.344420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5128.728453] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5132.565677] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5134.435660] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5136.811095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5136.912431] Lustre: 171960:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5137.004795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5137.015126] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 5138.866767] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5142.544365] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5143.997980] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5144.075255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5145.559516] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5145.630119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5147.619028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5147.621374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.622620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5147.626970] Lustre: Skipped 2 previous similar messages [ 5147.629440] Lustre: Skipped 1 previous similar message [ 5148.371240] Lustre: server umount lustre-OST0000 complete [ 5155.733403] Lustre: server umount lustre-MDT0000 complete [ 5156.959082] LustreError: 169985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787514168 with bad export cookie 18315477882679516634 [ 5156.961945] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5156.963086] LustreError: 169985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5157.076936] Lustre: server umount lustre-MDT0001 complete [ 5160.148607] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:42:51 (1787514171) [ 5162.379328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5162.520183] LustreError: 174369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5162.527450] LustreError: 174369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 5163.780930] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5166.587358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5168.082530] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5170.623244] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5172.377027] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5174.614045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5176.698233] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5179.015080] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5193.586355] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:43:24 (1787514204) [ 5197.280094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5197.283546] LustreError: Skipped 1 previous similar message [ 5197.285577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.290875] Lustre: Skipped 2 previous similar messages [ 5197.293270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5197.295926] Lustre: Skipped 3 previous similar messages [ 5200.665071] Lustre: server umount lustre-OST0000 complete [ 5202.399904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5202.402882] LustreError: Skipped 1 previous similar message [ 5216.223148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5216.282928] Lustre: server umount lustre-MDT0000 complete [ 5217.493294] LustreError: 174352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787514228 with bad export cookie 18315477882679517502 [ 5217.496495] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5217.497080] LustreError: 174352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5217.625988] Lustre: server umount lustre-MDT0001 complete [ 5221.210579] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 5222.259770] Key type lgssc unregistered [ 5222.389357] LNet: 178119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5222.391965] LNetError: 178119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5222.400267] LNet: Removed LNI 192.168.203.144@tcp [ 5222.701509] Key type .llcrypt unregistered [ 5222.702773] Key type ._llcrypt unregistered [ 5230.028390] Key type ._llcrypt registered [ 5230.029942] Key type .llcrypt registered [ 5230.071388] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5230.476742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5230.492489] alg: No test for adler32 (adler32-zlib) [ 5231.350326] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 5231.444801] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 5233.031201] Key type lgssc registered [ 5233.387589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5236.133071] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5236.139346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5251.551384] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5256.671294] LustreError: 179338:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.144@tcp: failed processing log, type 4: rc = -110 [ 5261.791456] LustreError: 179338:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5261.796054] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5261.799171] LustreError: MGC192.168.203.144@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5261.804815] LustreError: 179338:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5261.809203] LustreError: 179338:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 5261.811957] LustreError: 179338:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 5261.831084] Lustre: server umount lustre-OST0000 complete [ 5261.833165] LustreError: 179338:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5264.697041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5265.838756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5267.180883] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5269.987541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5270.108679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5271.492714] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5273.951940] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5275.872845] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5278.256840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5278.331880] LustreError: 181650:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5278.337278] Lustre: 181650:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.144@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5278.390123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5280.465265] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5282.968985] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5285.344025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5285.346990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.352432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5285.860094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.861279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5285.865506] Lustre: Skipped 1 previous similar message [ 5285.869857] Lustre: Skipped 1 previous similar message [ 5289.807798] Lustre: server umount lustre-MDT0000 complete [ 5290.976993] LustreError: 180576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.982959] LustreError: 180576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5291.014966] LustreError: 181679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787514302 with bad export cookie 10097995725866855787 [ 5291.018093] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5291.019075] LustreError: 181679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5291.114421] Lustre: server umount lustre-MDT0001 complete [ 5302.552557] Lustre: server umount lustre-OST0000 complete [ 5305.489774] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 5308.812718] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5321.014834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5321.103513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5321.113871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5321.151066] Lustre: lustre-MDT0000: new disk, initializing [ 5321.172857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5321.179151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5322.532105] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5326.840415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5326.885781] Lustre: 185168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5326.916218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5326.919220] Lustre: Skipped 1 previous similar message [ 5326.954386] Lustre: lustre-MDT0001: new disk, initializing [ 5326.988251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5327.002689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5327.006850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5328.319073] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5332.050606] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5333.858440] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5336.154431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5336.256813] Lustre: lustre-OST0000: new disk, initializing [ 5336.259209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5336.263597] Lustre: 187137:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5336.292774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5338.218917] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5338.425825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5338.430335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5338.463854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5341.970840] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5343.375840] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5343.450761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5344.783504] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5344.853057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5348.835189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5348.835944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5348.838445] LustreError: Skipped 1 previous similar message [ 5348.839125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5348.843848] Lustre: Skipped 1 previous similar message [ 5351.634059] Lustre: server umount lustre-OST0000 complete [ 5352.927727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5352.930661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5352.936208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5352.937775] Lustre: Skipped 1 previous similar message [ 5359.072624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5359.075234] Lustre: Skipped 3 previous similar messages [ 5367.263096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5367.319637] Lustre: server umount lustre-MDT0000 complete [ 5368.568727] LustreError: 185961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787514379 with bad export cookie 10097995725866856487 [ 5368.571129] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5368.573422] LustreError: 185961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5368.696069] Lustre: server umount lustre-MDT0001 complete [ 5371.456770] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:46:22 (1787514382) [ 5376.973563] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 5377.963988] Key type lgssc unregistered [ 5378.085432] LNet: 190080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5378.088882] LNetError: 190080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5378.099272] LNet: Removed LNI 192.168.203.144@tcp [ 5378.400914] Key type .llcrypt unregistered [ 5378.402250] Key type ._llcrypt unregistered [ 5385.473422] Key type ._llcrypt registered [ 5385.474846] Key type .llcrypt registered [ 5385.515964] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5385.934534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5385.940954] alg: No test for adler32 (adler32-zlib) [ 5386.790900] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 5386.887957] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 5388.471117] Key type lgssc registered [ 5388.812070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5391.368215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5391.374990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5392.516147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5393.942323] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5396.699482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5396.820785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5398.128237] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5400.533521] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5402.265462] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5404.790665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5404.859433] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5404.861154] LustreError: 193134:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5404.864610] Lustre: 193134:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.144@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5404.901820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.803122] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5409.220757] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5409.899774] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5409.901137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5409.904132] Lustre: Skipped 2 previous similar messages [ 5409.906271] Lustre: Skipped 1 previous similar message [ 5415.392393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5415.394853] Lustre: Skipped 3 previous similar messages [ 5416.013757] Lustre: server umount lustre-MDT0000 complete [ 5417.247239] LustreError: 191298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787514428 with bad export cookie 16005712583308841913 [ 5417.250429] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5417.251893] LustreError: 191298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5417.353407] Lustre: server umount lustre-MDT0001 complete [ 5428.885142] Lustre: server umount lustre-OST0000 complete [ 5431.841179] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 5435.181421] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5447.436265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5447.523748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5447.535286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5447.571330] Lustre: lustre-MDT0000: new disk, initializing [ 5447.595232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5447.602597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5448.987382] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5453.146507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5453.189111] Lustre: 196655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5453.209295] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5453.211108] Lustre: Skipped 1 previous similar message [ 5453.212893] LustreError: 197436:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5453.216065] LustreError: 197436:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5453.219368] Lustre: 197436:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5453.228401] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5453.230855] Lustre: Skipped 1 previous similar message [ 5453.266881] Lustre: lustre-MDT0001: new disk, initializing [ 5453.302456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5453.317257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5453.322552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5454.704266] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5458.497229] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5460.317550] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5462.644285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5462.743861] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5462.745230] Lustre: Skipped 1 previous similar message [ 5462.746692] LustreError: 198597:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5462.750832] LustreError: 198597:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5462.753740] Lustre: 198597:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5462.757845] Lustre: 198597:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5462.768215] Lustre: lustre-OST0000: new disk, initializing [ 5462.770952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5462.774631] Lustre: 198621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5462.807427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.856422] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5468.742385] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5470.158831] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5472.757672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5472.761899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5472.792159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5473.286755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5474.770839] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5474.848177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5477.859510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5477.860183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5477.866225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5477.868725] LustreError: Skipped 1 previous similar message [ 5481.612628] Lustre: server umount lustre-OST0000 complete [ 5482.976491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5482.977161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5482.981645] Lustre: Skipped 2 previous similar messages [ 5482.984415] Lustre: Skipped 1 previous similar message [ 5488.096214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5488.098678] Lustre: Skipped 2 previous similar messages [ 5497.311172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5497.397142] Lustre: server umount lustre-MDT0000 complete [ 5498.336146] LustreError: 196665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5498.342540] LustreError: 196665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5498.631373] LustreError: 196647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787514509 with bad export cookie 16005712583308842592 [ 5498.634946] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5498.635773] LustreError: 196647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5498.746648] Lustre: server umount lustre-MDT0001 complete [ 5501.774792] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 15:48:32 (1787514512) [ 5507.486150] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5511.203511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5511.336212] LustreError: 201765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5511.349055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5512.710810] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5515.445709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5516.881160] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5517.794258] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5519.966126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5522.044574] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5525.030535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5525.099672] Lustre: lustre-OST0001: new disk, initializing [ 5525.102162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5525.105847] Lustre: 204070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5526.146425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5526.150852] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5526.188577] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5527.226138] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5532.704380] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5539.466631] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5540.145354] Lustre: DEBUG MARKER: ADD OST3 [ 5541.797990] loop: module loaded [ 5541.815461] loop0: detected capacity change from 0 to 204800000 [ 5541.893228] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5541.896912] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5541.901243] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5544.049357] loop0: detected capacity change from 0 to 204800000 [ 5545.934083] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5546.005888] Lustre: lustre-OST0003: new disk, initializing [ 5546.008169] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5546.011188] Lustre: 205802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5546.052744] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5546.056726] Lustre: Skipped 3 previous similar messages [ 5546.937883] Lustre: DEBUG MARKER: STOP OST3 [ 5547.621692] Lustre: Failing over lustre-OST0003 [ 5547.673818] Lustre: server umount lustre-OST0003 complete [ 5547.789641] LustreError: 203249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5548.275041] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5552.267638] Lustre: DEBUG MARKER: START OST3 again [ 5552.815273] LustreError: 203248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5552.822946] LustreError: 203248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5554.608310] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5554.699471] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5555.945170] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5555.957609] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5556.689860] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5559.788672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5559.792209] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5559.803115] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5570.230178] Lustre: Failing over lustre-OST0003 [ 5570.267595] Lustre: server umount lustre-OST0003 complete [ 5575.140972] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5575.142899] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5575.143993] LustreError: Skipped 1 previous similar message [ 5575.148705] LustreError: 204240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5575.154848] LustreError: 204240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5585.378320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5585.378814] LustreError: 204240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5585.380473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5585.380479] Lustre: Skipped 2 previous similar messages [ 5585.382461] Lustre: Skipped 4 previous similar messages [ 5585.394097] LustreError: 204240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5588.765179] Lustre: server umount lustre-MDT0000 complete [ 5589.916913] LustreError: 201746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787514600 with bad export cookie 16005712583308843467 [ 5589.920401] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5589.920746] LustreError: 201746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5604.319135] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5604.417077] Lustre: server umount lustre-MDT0001 complete [ 5605.668228] Lustre: server umount lustre-OST0000 complete [ 5606.935552] Lustre: server umount lustre-OST0001 complete [ 5609.093201] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 5611.581400] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5623.523880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.624021] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5623.635177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5623.676811] Lustre: lustre-MDT0000: new disk, initializing [ 5623.699290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5623.701897] Lustre: Skipped 1 previous similar message [ 5623.708299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5625.064148] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5629.127409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5629.166308] Lustre: 211209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5629.265525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5630.527096] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5634.164702] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5635.969770] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5638.256361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5638.363973] Lustre: 213178:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5640.314748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5640.317439] Lustre: Skipped 1 previous similar message [ 5640.318867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5640.326290] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5640.349982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5644.097879] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5645.643386] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5645.718939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5647.144481] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5647.222912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5650.912704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5650.913723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5650.916598] LustreError: Skipped 1 previous similar message [ 5650.919799] Lustre: Skipped 2 previous similar messages [ 5650.920571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5650.927106] Lustre: Skipped 10 previous similar messages [ 5662.175148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5662.230370] Lustre: server umount lustre-OST0000 complete [ 5665.249170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5669.716250] Lustre: server umount lustre-MDT0000 complete [ 5671.072298] LustreError: 211204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787514682 with bad export cookie 16005712583308845707 [ 5671.076394] LustreError: 211204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5671.077278] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5671.191701] Lustre: server umount lustre-MDT0001 complete [ 5672.435697] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 15:51:23 (1787514683) [ 5677.314669] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 5680.595074] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5692.022039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5692.105672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5692.117854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5692.120224] Lustre: Skipped 3 previous similar messages [ 5692.158571] Lustre: lustre-MDT0000: new disk, initializing [ 5692.160764] Lustre: Skipped 2 previous similar messages [ 5692.188728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5693.689540] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5698.000879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.043754] Lustre: 217777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5698.144865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5699.464526] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5703.089772] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5704.966819] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5707.221801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5707.319063] Lustre: 219747:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5709.019887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5709.213418] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5712.852968] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5714.283266] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5714.356887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5715.746498] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5715.824685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5719.523730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5719.524223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5719.526830] LustreError: Skipped 1 previous similar message [ 5719.527381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5719.532551] Lustre: Skipped 4 previous similar messages [ 5719.534262] Lustre: Skipped 7 previous similar messages [ 5722.578243] Lustre: server umount lustre-OST0000 complete [ 5729.948058] Lustre: server umount lustre-MDT0000 complete [ 5731.217156] LustreError: 217770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787514742 with bad export cookie 16005712583308846589 [ 5731.220029] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5731.220916] LustreError: 217770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5731.345928] Lustre: server umount lustre-MDT0001 complete [ 5734.278301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.426186] LustreError: 221962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5734.432731] LustreError: 221962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5735.887668] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5738.854768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.462637] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5743.033628] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5745.018161] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5747.306850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5749.254702] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5751.705859] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5770.209228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5770.209850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.211472] LustreError: Skipped 1 previous similar message [ 5770.216276] Lustre: Skipped 2 previous similar messages [ 5774.608595] Lustre: server umount lustre-OST0000 complete [ 5785.568063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5785.570412] Lustre: Skipped 11 previous similar messages [ 5790.175067] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5790.242594] Lustre: server umount lustre-MDT0000 complete [ 5790.688018] LustreError: 222690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.693342] LustreError: 222690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5791.510262] LustreError: 221942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787514802 with bad export cookie 16005712583308847471 [ 5791.512908] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5791.515201] LustreError: 221942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5791.628807] Lustre: server umount lustre-MDT0001 complete [ 5884.547932] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 5885.546164] Key type lgssc unregistered [ 5885.675348] LNet: 225648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5885.678933] LNetError: 225648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5885.688315] LNet: Removed LNI 192.168.203.144@tcp [ 5886.012119] Key type .llcrypt unregistered [ 5886.013395] Key type ._llcrypt unregistered [ 5894.335526] Key type ._llcrypt registered [ 5894.336874] Key type .llcrypt registered [ 5894.371376] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 5900.477331] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5900.749932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5900.759207] alg: No test for adler32 (adler32-zlib) [ 5901.610069] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 5901.695075] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 5903.279150] Key type lgssc registered [ 5903.640815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5915.083338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5915.089297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5916.167171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5916.177832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5916.211584] Lustre: lustre-MDT0000: new disk, initializing [ 5916.230099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5916.235673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5917.580237] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5921.672722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5921.711659] Lustre: 228973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5921.733475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5921.735634] Lustre: Skipped 1 previous similar message [ 5921.769172] Lustre: lustre-MDT0001: new disk, initializing [ 5921.801768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5921.813841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5921.817851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5923.106914] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5926.777627] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5928.587970] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5930.886918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5931.013414] Lustre: lustre-OST0000: new disk, initializing [ 5931.016251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5931.020292] Lustre: 230937:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5931.053413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5933.047491] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5936.803736] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5937.149055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5937.153947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5937.200666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5938.316335] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5938.394745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5939.846867] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5939.924874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5942.242058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5942.242386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5942.244297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5942.246346] LustreError: Skipped 1 previous similar message [ 5942.255825] Lustre: Skipped 1 previous similar message [ 5946.712795] Lustre: server umount lustre-OST0000 complete [ 5952.480775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5952.481070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5952.481514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5952.481519] Lustre: Skipped 1 previous similar message [ 5952.486344] Lustre: Skipped 1 previous similar message [ 5954.206547] Lustre: server umount lustre-MDT0000 complete [ 5955.459332] LustreError: 229009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787514966 with bad export cookie 7287321729451384186 [ 5955.462754] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5955.462932] LustreError: 229009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5955.570452] Lustre: server umount lustre-MDT0001 complete [ 5961.905596] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5965.401051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.560964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5966.955804] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5969.818337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5969.943227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5971.317412] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5972.308446] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5974.505392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5976.497942] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5979.411638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5979.482881] Lustre: lustre-OST0001: new disk, initializing [ 5979.485302] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5979.488760] Lustre: 236175:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5979.521904] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5979.524891] Lustre: Skipped 1 previous similar message [ 5980.862146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5980.866525] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5980.902996] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5981.618708] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5987.305308] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5990.053828] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5994.129326] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 15:56:45 (1787515005) [ 5995.489140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5995.492991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5995.499509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5995.501464] Lustre: Skipped 1 previous similar message [ 6001.053508] Lustre: server umount lustre-MDT0000 complete [ 6001.633521] LustreError: 233863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.639768] LustreError: 233863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6002.299142] LustreError: 233851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787515013 with bad export cookie 7287321729451385054 [ 6002.302444] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6002.303046] LustreError: 233851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6002.419234] Lustre: server umount lustre-MDT0001 complete [ 6013.951961] Lustre: server umount lustre-OST0000 complete [ 6025.434689] Lustre: server umount lustre-OST0001 complete [ 6027.539829] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 6030.664118] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6042.698717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6042.784066] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6042.796403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6042.830304] Lustre: lustre-MDT0000: new disk, initializing [ 6042.849414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6042.855285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6044.268671] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6048.389994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6048.433152] Lustre: 240114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6048.550510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6049.939437] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6053.661724] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6055.565793] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6057.774743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6057.896927] Lustre: lustre-OST0000: new disk, initializing [ 6057.898827] Lustre: Skipped 1 previous similar message [ 6057.900538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6057.902464] Lustre: Skipped 2 previous similar messages [ 6057.904551] Lustre: 242083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6059.785162] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6059.895366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6059.899057] Lustre: Skipped 1 previous similar message [ 6059.901374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6059.934294] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6063.486182] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6064.946333] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6065.019654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6066.393099] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6066.470477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6070.243978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6070.245167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.245950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6070.245958] Lustre: Skipped 4 previous similar messages [ 6070.246682] LustreError: Skipped 1 previous similar message [ 6070.252633] Lustre: Skipped 4 previous similar messages [ 6073.172614] Lustre: server umount lustre-OST0000 complete [ 6075.361924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6075.362922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6075.366628] Lustre: Skipped 1 previous similar message [ 6075.369181] Lustre: Skipped 1 previous similar message [ 6080.671211] Lustre: server umount lustre-MDT0000 complete [ 6081.974232] LustreError: 242082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787515093 with bad export cookie 7287321729451387147 [ 6081.977193] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6081.979316] LustreError: 242082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6082.100196] Lustre: server umount lustre-MDT0001 complete [ 6085.075301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6085.235612] LustreError: 244297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6085.260660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6085.263042] Lustre: Skipped 2 previous similar messages [ 6086.673195] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6089.504986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6091.004485] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6093.414329] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6095.304441] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6097.479402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6099.454711] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6101.705038] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6248.416345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6248.418629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6248.423809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6248.425636] Lustre: Skipped 2 previous similar messages [ 6260.191155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6260.253498] Lustre: server umount lustre-OST0000 complete [ 6263.776090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6263.779091] LustreError: Skipped 1 previous similar message [ 6264.800023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6264.800732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6264.801974] Lustre: Skipped 7 previous similar messages [ 6264.805333] Lustre: Skipped 2 previous similar messages [ 6267.601234] Lustre: server umount lustre-MDT0000 complete [ 6268.796099] LustreError: 244322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787515279 with bad export cookie 7287321729451388029 [ 6268.799827] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6268.800693] LustreError: 244322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6268.902023] Lustre: server umount lustre-MDT0001 complete [ 6367.883443] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 6368.919691] Key type lgssc unregistered [ 6369.049356] LNet: 247919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6369.052219] LNetError: 247919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6369.060380] LNet: Removed LNI 192.168.203.144@tcp [ 6369.394123] Key type .llcrypt unregistered [ 6369.395260] Key type ._llcrypt unregistered [ 6377.435754] Key type ._llcrypt registered [ 6377.436688] Key type .llcrypt registered [ 6377.474032] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 6383.573575] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6383.916036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6383.959044] alg: No test for adler32 (adler32-zlib) [ 6384.802301] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 6384.884614] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 6386.463113] Key type lgssc registered [ 6386.806156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6400.512539] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6403.648586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6403.655487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6404.729102] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6404.739170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6404.773435] Lustre: lustre-MDT0000: new disk, initializing [ 6404.791194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6404.797370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6406.056907] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6410.085878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6410.127636] Lustre: 251961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6410.148250] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6410.149895] Lustre: Skipped 1 previous similar message [ 6410.182534] Lustre: lustre-MDT0001: new disk, initializing [ 6410.213925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6410.224417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6410.229295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6411.541162] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6413.744078] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6415.769752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6415.862592] Lustre: lustre-OST0000: new disk, initializing [ 6415.864237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6415.867626] Lustre: 253589:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6415.890922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6416.759418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6416.762534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6416.804455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6417.691814] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6421.642756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6421.705219] Lustre: lustre-OST0001: new disk, initializing [ 6421.707389] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6421.710245] Lustre: 254463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6421.745568] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6423.564189] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6424.451194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6424.454861] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6424.496259] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6428.848477] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6432.380819] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6435.807903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6435.811064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.816518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6439.907050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6439.907332] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6439.913570] Lustre: Skipped 2 previous similar messages [ 6441.190709] Lustre: server umount lustre-MDT0000 complete [ 6442.358988] LustreError: 251952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787515453 with bad export cookie 288894946823795664 [ 6442.362322] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6442.363815] LustreError: 251952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6442.464221] Lustre: server umount lustre-MDT0001 complete [ 6453.251610] Lustre: server umount lustre-OST0000 complete [ 6464.728403] Lustre: server umount lustre-OST0001 complete [ 6469.020162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6471.961292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6474.557689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6477.435901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6483.153199] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6486.402813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6486.410529] Lustre: lustre-MDT0000: reset Object Index mappings [ 6486.550618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6487.811035] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6490.450528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6490.458743] Lustre: lustre-MDT0001: reset Object Index mappings [ 6491.854117] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6492.737368] Lustre: 260599:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6494.745802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6494.857360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6494.859248] Lustre: Skipped 1 previous similar message [ 6496.621516] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6496.930064] LustreError: 260953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6496.936048] LustreError: 260953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6496.936674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6499.257863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6500.391899] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6501.027077] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6504.022823] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6505.163688] Lustre: 262468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6510.561917] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6510.564877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6510.566882] Lustre: Skipped 2 previous similar messages [ 6510.570526] Lustre: Skipped 4 previous similar messages [ 6512.348520] Lustre: server umount lustre-MDT0000 complete [ 6513.647387] LustreError: 259441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787515524 with bad export cookie 288894946823802440 [ 6513.651036] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6513.653160] LustreError: 259441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6513.753204] Lustre: server umount lustre-MDT0001 complete [ 6525.884078] Lustre: server umount lustre-OST0000 complete [ 6537.306353] Lustre: server umount lustre-OST0001 complete [ 6542.870988] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 16:05:53 (1787515553) [ 6548.258542] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 6551.420844] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6563.808026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6563.893923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6563.904641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6563.946923] Lustre: lustre-MDT0000: new disk, initializing [ 6563.967645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6563.969737] Lustre: Skipped 1 previous similar message [ 6563.975091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6565.324880] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6569.569676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6569.610286] Lustre: 266712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6569.713613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6571.229461] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6575.040522] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6576.834409] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6579.063600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6579.165036] Lustre: lustre-OST0000: new disk, initializing [ 6579.166664] Lustre: Skipped 1 previous similar message [ 6579.168440] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6579.170823] Lustre: Skipped 2 previous similar messages [ 6579.173188] Lustre: 268679:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6580.409187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6580.413582] Lustre: Skipped 1 previous similar message [ 6580.415573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6580.446212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6581.153692] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6584.856423] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6586.399898] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6586.479866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6587.988753] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6588.076089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6590.945028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6590.945325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6590.948185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6590.948784] LustreError: Skipped 1 previous similar message [ 6590.958022] Lustre: Skipped 2 previous similar messages [ 6596.066147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6596.068271] Lustre: Skipped 2 previous similar messages [ 6603.231121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6603.283769] Lustre: server umount lustre-OST0000 complete [ 6605.791933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6605.796526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6605.802566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6605.804122] Lustre: Skipped 2 previous similar messages [ 6610.666601] Lustre: server umount lustre-MDT0000 complete [ 6611.424102] LustreError: 266722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6611.428739] LustreError: 266722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6611.921670] LustreError: 266704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787515622 with bad export cookie 288894946823804512 [ 6611.927533] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6611.927799] LustreError: 266704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6612.059248] Lustre: server umount lustre-MDT0001 complete [ 6618.218211] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6621.702541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6621.835584] LustreError: 271611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6621.848271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6621.851207] Lustre: Skipped 2 previous similar messages [ 6623.253110] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6626.178263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.722758] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6628.623761] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6630.739783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6632.833455] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6635.600918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6635.669594] Lustre: lustre-OST0001: new disk, initializing [ 6635.672145] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6635.676176] Lustre: 273917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6636.756293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6636.759882] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6636.808061] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6637.760282] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6642.143892] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6643.384491] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6645.239748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6645.245195] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6647.267327] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6647.268764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6647.271351] Lustre: Skipped 3 previous similar messages [ 6647.274718] Lustre: Skipped 5 previous similar messages [ 6653.081060] Lustre: server umount lustre-MDT0000 complete [ 6654.337273] LustreError: 271594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787515665 with bad export cookie 288894946823805394 [ 6654.340453] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6654.340639] LustreError: 271594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6654.461969] Lustre: server umount lustre-MDT0001 complete [ 6665.734643] Lustre: server umount lustre-OST0000 complete [ 6677.277668] Lustre: server umount lustre-OST0001 complete [ 6682.779442] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6686.470389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6686.625730] LustreError: 276972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6686.645965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6686.649258] Lustre: Skipped 3 previous similar messages [ 6688.117876] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6690.893849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6692.495022] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6693.426547] Lustre: 278114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6695.630317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6696.804317] LustreError: 278467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6696.813442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6697.755556] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6700.650465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6701.751752] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6702.758388] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6705.926928] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6707.146855] Lustre: 279990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6708.932535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6708.938658] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6712.096892] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 16:08:43 (1787515723) [ 6727.136286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6727.140089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6727.143711] Lustre: Skipped 1 previous similar message [ 6727.145675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6727.148210] Lustre: Skipped 4 previous similar messages [ 6729.754832] Lustre: server umount lustre-MDT0000 complete [ 6731.007110] LustreError: 276954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787515742 with bad export cookie 288894946823811701 [ 6731.010513] LustreError: 276954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6731.010837] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6732.768199] LustreError: 277714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6732.774072] LustreError: 277714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6737.211705] Lustre: server umount lustre-MDT0001 complete [ 6744.576872] Lustre: server umount lustre-OST0000 complete [ 6751.955553] Lustre: server umount lustre-OST0001 complete [ 6754.009654] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 6757.094582] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6771.552680] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6774.995313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6775.074896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6775.084800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6775.118426] Lustre: lustre-MDT0000: new disk, initializing [ 6775.141973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6776.654675] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6780.994282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6781.044427] Lustre: 284217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6781.150525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6782.632528] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6787.293439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6787.437576] Lustre: 285851:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6788.393807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6789.657653] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6794.023919] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6795.253257] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6796.155603] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6800.704702] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6806.040796] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 16:10:17 (1787515817) [ 6807.008048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6807.011914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6807.017910] Lustre: Skipped 5 previous similar messages [ 6807.020941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6807.024016] Lustre: Skipped 7 previous similar messages [ 6812.894105] Lustre: server umount lustre-MDT0000 complete [ 6814.121353] LustreError: 284255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787515825 with bad export cookie 288894946823819569 [ 6814.124736] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6814.125116] LustreError: 284255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6814.243891] Lustre: server umount lustre-MDT0001 complete [ 6825.920892] Lustre: server umount lustre-OST0000 complete [ 6837.407283] Lustre: server umount lustre-OST0001 complete [ 6840.327055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.494447] LustreError: 288963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6840.500042] LustreError: 288963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6840.526521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6840.528458] Lustre: Skipped 7 previous similar messages [ 6841.921651] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6844.847446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6846.391849] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6848.881322] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6850.700090] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6852.851668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6854.814852] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6857.075829] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6872.741779] LustreError: 290792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6872.746985] LustreError: 290792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6873.447686] loop1: detected capacity change from 0 to 204800000 [ 6873.525547] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6873.529976] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6873.534195] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6875.142222] loop1: detected capacity change from 0 to 204800000 [ 6877.010656] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.042417] LustreError: 288953:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs10330: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6877.047309] LustreError: 288953:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs10330 [ 6877.049448] LustreError: 288953:0:(mgs_handler.c:517:mgs_target_reg()) fs10330-MDT0001: Error checking target index: rc = -13 [ 6877.052943] LustreError: fs10330-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6877.056343] LustreError: 292697:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 6877.058933] LustreError: 292697:0:(tgt_mount.c:2132:server_put_super()) no obd fs10330-MDT0001 [ 6877.061277] LustreError: 292697:0:(tgt_mount.c:116:server_deregister_mount()) fs10330-MDT0001 not registered [ 6877.072470] Lustre: server umount fs10330-MDT0001 complete [ 6877.074096] LustreError: 292697:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6879.860494] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6879.890716] Lustre: Setting parameter fs10330-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs10330-MDT0001 [ 6879.894138] Lustre: Skipped 2 previous similar messages [ 6879.908910] Lustre: srv-fs10330-MDT0001: No data found on store. Initialize space. [ 6879.911250] Lustre: Skipped 4 previous similar messages [ 6879.953225] Lustre: fs10330-MDT0001: new disk, initializing [ 6879.955244] Lustre: Skipped 3 previous similar messages [ 6879.988174] Lustre: srv-fs10330-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6879.990642] Lustre: 293162:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10330-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6879.993420] LustreError: 293162:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10330-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6881.319600] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6884.189223] Lustre: server umount fs10330-MDT0001 complete [ 6888.434455] loop1: detected capacity change from 0 to 204800000 [ 6890.257638] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6890.375134] Lustre: 294729:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10330-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6890.378677] LustreError: 294729:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10330-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6891.707830] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6893.308677] Lustre: server umount fs10330-MDT0001 complete [ 6897.121446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6897.126034] Lustre: Skipped 5 previous similar messages [ 6902.623861] Lustre: server umount lustre-MDT0000 complete [ 6905.329174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6905.368210] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6906.873727] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6909.327253] loop1: detected capacity change from 0 to 204800000 [ 6910.952106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6911.163622] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6911.282703] Lustre: 297171:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10330-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6911.286545] LustreError: 297171:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10330-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6912.780461] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6914.773228] Lustre: server umount fs10330-MDT0001 complete [ 6921.696025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6927.694889] Lustre: server umount lustre-OST0000 complete [ 6936.544806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6936.546689] Lustre: Skipped 17 previous similar messages [ 6943.199079] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6943.262173] Lustre: server umount lustre-MDT0000 complete [ 6944.637969] LustreError: 295220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787515955 with bad export cookie 288894946823822243 [ 6944.642988] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6944.644859] LustreError: 295220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6944.775873] Lustre: server umount lustre-MDT0001 complete [ 6948.600191] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 6949.584768] Key type lgssc unregistered [ 6949.716408] LNet: 299514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6949.720352] LNetError: 299514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6949.727709] LNet: Removed LNI 192.168.203.144@tcp [ 6950.081116] Key type .llcrypt unregistered [ 6950.083330] Key type ._llcrypt unregistered [ 6957.340955] Key type ._llcrypt registered [ 6957.342046] Key type .llcrypt registered [ 6957.378022] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6957.749149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6957.764395] alg: No test for adler32 (adler32-zlib) [ 6958.643204] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 6958.733122] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 6960.319141] Key type lgssc registered [ 6960.751143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6963.457397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6963.463374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6964.620086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6966.074374] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6968.846962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6968.976159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6970.257186] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6972.434102] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6974.080569] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6976.216746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6976.354039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6978.237981] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6979.429080] LustreError: 302589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6979.434987] LustreError: 302589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6981.209834] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6984.675505] LustreError: 302590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6989.798021] LustreError: 303074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6989.803385] LustreError: 303074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6994.917262] LustreError: 302589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.921688] LustreError: 302589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6997.367816] loop1: detected capacity change from 0 to 204800000 [ 6997.431917] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6997.436055] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6997.439877] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6999.113796] loop1: detected capacity change from 0 to 204800000 [ 7000.036757] LustreError: 302589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7000.041062] LustreError: 302589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7000.903748] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7000.953096] Lustre: fs29583-OST0000: new disk, initializing [ 7000.955443] Lustre: srv-fs29583-OST0000: No data found on store. Initialize space. [ 7000.958042] Lustre: 304515:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7000.979189] Lustre: fs29583-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7002.847305] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7006.188302] Lustre: server umount fs29583-OST0000 complete [ 7010.147127] loop1: detected capacity change from 0 to 204800000 [ 7010.278725] LustreError: 302589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7010.282870] LustreError: 302589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7011.985775] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7012.049298] Lustre: fs29583-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7013.865596] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7016.577037] loop2: detected capacity change from 0 to 204800000 [ 7016.654563] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7016.658310] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7016.662391] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7018.294213] loop2: detected capacity change from 0 to 204800000 [ 7020.237552] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7020.272153] LustreError: fs29583-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 7020.276222] LustreError: fs29583-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7020.278886] LustreError: 307551:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 7020.281098] LustreError: 307551:0:(tgt_mount.c:2132:server_put_super()) no obd fs29583-OST0001 [ 7020.282931] LustreError: 307551:0:(tgt_mount.c:116:server_deregister_mount()) fs29583-OST0001 not registered [ 7020.292700] Lustre: server umount fs29583-OST0001 complete [ 7020.294163] LustreError: 307551:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7023.496241] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7023.538553] Lustre: fs29583-OST0001: new disk, initializing [ 7023.540256] Lustre: srv-fs29583-OST0001: No data found on store. Initialize space. [ 7023.544443] Lustre: 308102:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7023.567071] Lustre: fs29583-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7025.289802] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7027.364579] LustreError: 303074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7027.369129] LustreError: 303074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7028.582267] Lustre: server umount fs29583-OST0000 complete [ 7031.566079] Lustre: server umount fs29583-OST0001 complete [ 7035.635610] loop1: detected capacity change from 0 to 204800000 [ 7037.529762] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7039.366590] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7041.393992] loop2: detected capacity change from 0 to 204800000 [ 7043.187343] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7043.243498] Lustre: fs29583-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7043.245553] Lustre: Skipped 1 previous similar message [ 7045.076828] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7047.900642] Lustre: server umount fs29583-OST0000 complete [ 7050.872908] Lustre: server umount fs29583-OST0001 complete [ 7056.359221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7056.365655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7056.365828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7060.565270] Lustre: server umount lustre-OST0000 complete [ 7066.594078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7066.594302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7066.595120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7066.595129] Lustre: Skipped 1 previous similar message [ 7066.602429] Lustre: Skipped 2 previous similar messages [ 7071.713659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7071.715562] Lustre: Skipped 2 previous similar messages [ 7076.319083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7076.394800] Lustre: server umount lustre-MDT0000 complete [ 7076.832253] LustreError: 300781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7076.836846] LustreError: 300781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 7077.632098] LustreError: 308101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787516088 with bad export cookie 10221038490583691296 [ 7077.635489] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7077.637276] LustreError: 308101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7077.753687] Lustre: server umount lustre-MDT0001 complete [ 7081.117457] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7082.094663] Key type lgssc unregistered [ 7082.224485] LNet: 314015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7082.226742] LNetError: 314015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7082.237409] LNet: Removed LNI 192.168.203.144@tcp [ 7082.572105] Key type .llcrypt unregistered [ 7082.573286] Key type ._llcrypt unregistered [ 7092.618435] Key type ._llcrypt registered [ 7092.620350] Key type .llcrypt registered [ 7092.657989] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7098.671379] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7099.112178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7099.161633] alg: No test for adler32 (adler32-zlib) [ 7100.035246] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 7100.138771] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7101.727196] Key type lgssc registered [ 7102.110949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7113.869795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7113.875251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7114.993289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7115.011056] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7115.068130] Lustre: lustre-MDT0000: new disk, initializing [ 7115.092155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7115.099152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7116.627461] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7120.909174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7120.951553] Lustre: 317852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7120.974682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7120.976462] Lustre: Skipped 1 previous similar message [ 7121.014487] Lustre: lustre-MDT0001: new disk, initializing [ 7121.047502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7121.058318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7121.061243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7122.355927] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7126.557983] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7128.680424] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7131.833982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7131.968100] Lustre: lustre-OST0000: new disk, initializing [ 7131.970412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7131.973524] Lustre: 319820:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7132.014542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7134.371090] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7138.499034] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7139.684168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7139.687172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7139.715440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7144.955453] Lustre: Mounted lustre-client - version 2.17.57_80_gd2e8623 [ 7145.425267] Lustre: Unmounted lustre-client [ 7150.050579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7150.055837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7150.061460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7150.071283] Lustre: Skipped 1 previous similar message [ 7151.529480] Lustre: server umount lustre-OST0000 complete [ 7155.168808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7155.170832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7155.177842] Lustre: Skipped 2 previous similar messages [ 7159.471172] Lustre: server umount lustre-MDT0000 complete [ 7161.197260] LustreError: 317889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787516172 with bad export cookie 3233595189783974471 [ 7161.200620] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7161.206263] LustreError: 317889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7161.370505] Lustre: server umount lustre-MDT0001 complete [ 7165.987240] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7167.309970] Key type lgssc unregistered [ 7193.119295] LNet: 321978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7193.121948] LNetError: 321978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7271.974576] LNet: Removed LNI 192.168.203.144@tcp [ 7272.348531] Key type .llcrypt unregistered [ 7272.349655] Key type ._llcrypt unregistered [ 7279.185828] Key type ._llcrypt registered [ 7279.187524] Key type .llcrypt registered [ 7279.225239] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7286.111135] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7286.528787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7286.563944] alg: No test for adler32 (adler32-zlib) [ 7287.441990] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 7287.541811] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7289.127161] Key type lgssc registered [ 7289.521844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7300.325044] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:18:31 (1787516311) [ 7302.742213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7302.752207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.884327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7303.898899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7303.939307] Lustre: lustre-MDT0000: new disk, initializing [ 7303.964244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7303.971592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7305.490061] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7309.800926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7309.845709] Lustre: 325174:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7309.869788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7309.872606] Lustre: Skipped 1 previous similar message [ 7309.910755] Lustre: lustre-MDT0001: new disk, initializing [ 7309.948220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7309.963167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7309.966457] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7311.390648] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7315.317391] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7317.177711] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7319.420018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7319.545796] Lustre: lustre-OST0000: new disk, initializing [ 7319.547629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7319.549685] Lustre: 327144:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7319.577098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7321.469710] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7322.998371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7323.002191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7323.034763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7324.990360] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7332.493645] Lustre: server umount lustre-OST0000 complete [ 7335.903864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7335.906202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7335.910020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7338.464535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7338.464730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7343.584188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7343.586050] Lustre: Skipped 2 previous similar messages [ 7343.613770] Lustre: server umount lustre-MDT0000 complete [ 7344.824395] LustreError: 325211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787516355 with bad export cookie 4440802480788127946 [ 7344.829415] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7354.382755] Lustre: server umount lustre-MDT0001 complete [ 7357.890906] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7358.861881] Key type lgssc unregistered [ 7358.989332] LNet: 329051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7358.991506] LNetError: 329051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7358.998266] LNet: Removed LNI 192.168.203.144@tcp [ 7359.287920] Key type .llcrypt unregistered [ 7359.288794] Key type ._llcrypt unregistered [ 7366.948514] Key type ._llcrypt registered [ 7366.949475] Key type .llcrypt registered [ 7366.988462] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7367.486333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7367.504589] alg: No test for adler32 (adler32-zlib) [ 7368.349340] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 7368.435460] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7370.015126] Key type lgssc registered [ 7370.385748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7372.844232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7372.851078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7374.001161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7375.282190] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7377.953737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7378.075688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7379.407709] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7381.648799] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7383.323027] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7385.410721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7385.529969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7387.365840] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7389.577979] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7403.227760] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7403.714733] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7405.027778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7405.028789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7405.031439] Lustre: Skipped 1 previous similar message [ 7407.064503] Lustre: server umount lustre-MDT0000 complete [ 7408.303128] LustreError: 330282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787516419 with bad export cookie 18247609604403102918 [ 7408.306584] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7408.307492] LustreError: 330282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7410.143868] LustreError: 332792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7410.144199] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7410.149085] LustreError: 332792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7410.153055] Lustre: Skipped 1 previous similar message [ 7410.153701] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7410.159117] Lustre: Skipped 3 previous similar messages [ 7413.151920] LustreError: 332792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7413.152166] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7413.156651] LustreError: 332792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7413.158526] Lustre: Skipped 1 previous similar message [ 7414.510629] Lustre: server umount lustre-MDT0001 complete [ 7430.111124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7430.175928] Lustre: server umount lustre-OST0000 complete [ 7433.164033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7434.606477] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7435.549602] Lustre: 334501:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.144@tcp [ 7436.223939] Lustre: server umount MGS complete [ 7439.047195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7439.218731] LustreError: 335072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7439.240720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7440.573869] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7443.417517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7443.547267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7444.913367] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7447.779345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7449.918985] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7452.343892] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7452.867965] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7454.706097] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7461.340573] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 7463.393461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.394953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7463.397701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7463.401416] Lustre: Skipped 1 previous similar message [ 7468.514425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7468.516300] Lustre: Skipped 1 previous similar message [ 7468.561381] Lustre: server umount lustre-OST0000 complete [ 7473.632670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7473.638416] Lustre: Skipped 1 previous similar message [ 7478.752093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7478.755201] Lustre: Skipped 3 previous similar messages [ 7484.383123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7484.449817] Lustre: server umount lustre-MDT0000 complete [ 7485.747679] LustreError: 335052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787516496 with bad export cookie 18247609604403103744 [ 7485.751851] LustreError: 335052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7485.753311] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7485.861350] Lustre: server umount lustre-MDT0001 complete [ 7489.234191] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7490.205607] Key type lgssc unregistered [ 7490.340419] LNet: 338510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7490.344319] LNetError: 338510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7490.352321] LNet: Removed LNI 192.168.203.144@tcp [ 7490.639376] Key type .llcrypt unregistered [ 7490.640580] Key type ._llcrypt unregistered [ 7496.319877] Key type ._llcrypt registered [ 7496.320967] Key type .llcrypt registered [ 7496.351350] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7502.143593] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7502.538762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7502.548301] alg: No test for adler32 (adler32-zlib) [ 7503.399064] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 7503.481869] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7505.063125] Key type lgssc registered [ 7505.398521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7516.682823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7516.688301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7517.802989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7517.813331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7517.845612] Lustre: lustre-MDT0000: new disk, initializing [ 7517.863770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7517.869050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7519.094494] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7523.111898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7523.156328] Lustre: 341643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7523.182807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7523.184846] Lustre: Skipped 1 previous similar message [ 7523.224665] Lustre: lustre-MDT0001: new disk, initializing [ 7523.258430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7523.271107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7523.274720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7524.650039] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7528.299844] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7530.036169] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7532.021169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7532.121157] Lustre: lustre-OST0000: new disk, initializing [ 7532.122699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7532.125892] Lustre: 343611:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7532.149506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7533.848534] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7537.250481] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7538.458093] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7540.725883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7540.729253] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7540.756338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7541.568943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 7542.742969] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7542.803672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7545.825410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7545.825550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.827689] LustreError: Skipped 1 previous similar message [ 7545.832179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7545.834718] Lustre: Skipped 1 previous similar message [ 7549.454898] Lustre: server umount lustre-OST0000 complete [ 7550.944935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7550.945578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7550.949220] Lustre: Skipped 2 previous similar messages [ 7556.064756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7556.066544] Lustre: Skipped 2 previous similar messages [ 7561.184584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7561.186853] Lustre: Skipped 1 previous similar message [ 7565.279075] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7565.344964] Lustre: server umount lustre-MDT0000 complete [ 7566.626176] LustreError: 341634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787516577 with bad export cookie 11722665892148974085 [ 7566.629674] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7566.630014] LustreError: 341634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7566.742223] Lustre: server umount lustre-MDT0001 complete [ 7569.578951] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 16:23:00 (1787516580) [ 7574.985657] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7578.383933] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7590.553249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7590.654689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7590.666495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7590.711306] Lustre: lustre-MDT0000: new disk, initializing [ 7590.735145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7590.742260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7592.178083] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7596.341238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7596.381644] Lustre: 348384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7596.404344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7596.407110] Lustre: Skipped 1 previous similar message [ 7596.447240] Lustre: lustre-MDT0001: new disk, initializing [ 7596.480112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7596.491900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7596.496078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7597.801225] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7601.516438] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7603.356533] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7605.658609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7605.776174] Lustre: lustre-OST0000: new disk, initializing [ 7605.778167] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7605.780946] Lustre: 350348:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7605.816187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7607.713136] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7611.382522] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7615.991290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7615.994487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7616.024837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7626.210773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7626.211223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7626.212159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7626.213179] LustreError: Skipped 1 previous similar message [ 7626.224052] Lustre: Skipped 1 previous similar message [ 7628.112818] Lustre: server umount lustre-OST0000 complete [ 7631.330073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7631.334701] Lustre: Skipped 1 previous similar message [ 7635.548518] Lustre: server umount lustre-MDT0000 complete [ 7636.448127] LustreError: 349184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7636.454285] LustreError: 349184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7636.898332] LustreError: 351460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787516647 with bad export cookie 11722665892148974967 [ 7636.901140] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7636.902818] LustreError: 351460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7637.034444] Lustre: server umount lustre-MDT0001 complete [ 7640.859673] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7641.967063] Key type lgssc unregistered [ 7642.111422] LNet: 352501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7642.113979] LNetError: 352501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7642.122338] LNet: Removed LNI 192.168.203.144@tcp [ 7642.445288] Key type .llcrypt unregistered [ 7642.447058] Key type ._llcrypt unregistered [ 7675.020683] Key type ._llcrypt registered [ 7675.021699] Key type .llcrypt registered [ 7675.056742] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7675.165240] Key type .llcrypt unregistered [ 7675.166314] Key type ._llcrypt unregistered [ 7701.808065] Key type ._llcrypt registered [ 7701.810941] Key type .llcrypt registered [ 7701.884917] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7702.012407] Key type .llcrypt unregistered [ 7702.013974] Key type ._llcrypt unregistered [ 7713.383867] Key type ._llcrypt registered [ 7713.385232] Key type .llcrypt registered [ 7713.426749] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7713.539476] Key type .llcrypt unregistered [ 7713.540707] Key type ._llcrypt unregistered [ 7724.662476] Key type ._llcrypt registered [ 7724.663424] Key type .llcrypt registered [ 7724.693807] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7724.790262] Key type .llcrypt unregistered [ 7724.791280] Key type ._llcrypt unregistered [ 7751.013524] Key type ._llcrypt registered [ 7751.014988] Key type .llcrypt registered [ 7751.062460] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7751.189467] Key type .llcrypt unregistered [ 7751.191046] Key type ._llcrypt unregistered [ 7762.333407] Key type ._llcrypt registered [ 7762.335098] Key type .llcrypt registered [ 7762.373171] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7762.493255] Key type .llcrypt unregistered [ 7762.494721] Key type ._llcrypt unregistered [ 7773.763316] Key type ._llcrypt registered [ 7773.765176] Key type .llcrypt registered [ 7773.810219] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7773.932502] Key type .llcrypt unregistered [ 7773.934679] Key type ._llcrypt unregistered [ 7785.161692] Key type ._llcrypt registered [ 7785.162710] Key type .llcrypt registered [ 7785.196139] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7785.305212] Key type .llcrypt unregistered [ 7785.306308] Key type ._llcrypt unregistered [ 7795.737599] Key type ._llcrypt registered [ 7795.738603] Key type .llcrypt registered [ 7795.772831] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7795.868178] Key type .llcrypt unregistered [ 7795.869176] Key type ._llcrypt unregistered [ 7806.277863] Key type ._llcrypt registered [ 7806.278872] Key type .llcrypt registered [ 7806.315036] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7806.422604] Key type .llcrypt unregistered [ 7806.423666] Key type ._llcrypt unregistered [ 7826.899315] Key type ._llcrypt registered [ 7826.900482] Key type .llcrypt registered [ 7826.939801] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7827.040260] Key type .llcrypt unregistered [ 7827.041601] Key type ._llcrypt unregistered [ 7837.528959] Key type ._llcrypt registered [ 7837.530127] Key type .llcrypt registered [ 7837.563115] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7837.665214] Key type .llcrypt unregistered [ 7837.666770] Key type ._llcrypt unregistered [ 7848.388864] Key type ._llcrypt registered [ 7848.390041] Key type .llcrypt registered [ 7848.430834] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7848.559268] Key type .llcrypt unregistered [ 7848.560673] Key type ._llcrypt unregistered [ 7866.630974] Key type ._llcrypt registered [ 7866.632902] Key type .llcrypt registered [ 7866.675595] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7872.984574] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7873.302947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7873.309364] alg: No test for adler32 (adler32-zlib) [ 7874.167067] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 7874.255350] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7875.831117] Key type lgssc registered [ 7876.233950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7887.993204] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7887.999343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7889.093606] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 7889.106112] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 7889.148054] Lustre: tst250fs-MDT0000: new disk, initializing [ 7889.172829] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7889.179659] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7890.668953] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7894.884055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7894.935173] Lustre: 372455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7894.962143] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 7894.964415] Lustre: Skipped 1 previous similar message [ 7895.004240] Lustre: tst250fs-MDT0001: new disk, initializing [ 7895.038647] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7895.054986] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7895.059371] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7896.446125] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7898.759163] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7900.947192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7901.063014] Lustre: tst250fs-OST0000: new disk, initializing [ 7901.065617] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 7901.069327] Lustre: 374087:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7901.100042] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7902.997506] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7907.028867] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7907.086143] Lustre: tst250fs-OST0001: new disk, initializing [ 7907.087823] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 7907.090331] Lustre: 374949:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7907.126705] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7908.163474] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7908.167961] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7908.171205] Lustre: Skipped 1 previous similar message [ 7908.234786] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7908.237324] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7908.947357] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7911.444140] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 7923.682945] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7923.684449] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7923.689290] Lustre: Skipped 1 previous similar message [ 7927.774762] Lustre: server umount tst250fs-MDT0000 complete [ 7928.953337] LustreError: 372448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787516940 with bad export cookie 13791188390782493251 [ 7928.957251] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7928.960622] LustreError: 372448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7929.064286] Lustre: server umount tst250fs-MDT0001 complete [ 7940.100210] Lustre: server umount tst250fs-OST0000 complete [ 7951.646145] Lustre: server umount tst250fs-OST0001 complete [ 7957.978748] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7960.704623] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7971.163410] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:29:42 (1787516982) [ 7971.678754] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 7977.406743] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7980.862973] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7990.694079] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7726 sec ========== 16:30:01 (1787517001) [ 7991.188879] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:30:02 (1787517002) === [ 7992.243700] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:30:03 (1787517003) === [ 8005.574575] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 8006.703311] Key type lgssc unregistered [ 8006.833371] LNet: 384070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8006.835753] LNetError: 384070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8006.845308] LNet: Removed LNI 192.168.203.144@tcp [ 8007.174134] Key type .llcrypt unregistered [ 8007.175779] Key type ._llcrypt unregistered