[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 688279199 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002000] kvm-guest: setup PV IPIs [ 0.002000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.002000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.002000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.002000] pid_max: default: 32768 minimum: 301 [ 0.002288] LSM: Security Framework initializing [ 0.003067] Yama: becoming mindful. [ 0.004037] SELinux: Initializing. [ 0.006054] *** VALIDATE selinux *** [ 0.018231] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026445] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027208] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029047] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031074] *** VALIDATE tmpfs *** [ 0.033098] *** VALIDATE proc *** [ 0.034228] *** VALIDATE cgroup *** [ 0.036006] *** VALIDATE cgroup2 *** [ 0.037273] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040032] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.045038] debug: unmapping init [mem 0xffffffff84259000-0xffffffff84260fff] [ 0.048456] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049972] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051020] ... version: 2 [ 0.052008] ... bit width: 48 [ 0.053012] ... generic registers: 4 [ 0.054017] ... value mask: 0000ffffffffffff [ 0.055009] ... max period: 00007fffffffffff [ 0.056024] ... fixed-purpose events: 3 [ 0.057608] ... event mask: 000000070000000f [ 0.058288] rcu: Hierarchical SRCU implementation. [ 0.062887] smp: Bringing up secondary CPUs ... [ 0.064825] x86: Booting SMP configuration: [ 0.065046] .... node #0, CPUs: #1 #2 #3 [ 0.077015] smp: Brought up 1 node, 4 CPUs [ 0.079474] smpboot: Max logical packages: 1 [ 0.080010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.161018] node 0 deferred pages initialised in 77ms [ 0.165158] devtmpfs: initialized [ 0.166147] x86/mm: Memory block size: 128MB [ 0.168955] gcov: version magic: 0x41383552 [ 0.170402] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.171065] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.172306] pinctrl core: initialized pinctrl subsystem [ 0.173151] [ 0.173657] ************************************************************* [ 0.174009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175009] ** ** [ 0.176010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.177009] ** ** [ 0.178010] ** This means that this kernel is built to expose internal ** [ 0.179010] ** IOMMU data structures, which may compromise security on ** [ 0.180010] ** your system. ** [ 0.181021] ** ** [ 0.182053] ** If you see this message and you are not debugging the ** [ 0.183008] ** kernel, report this immediately to your vendor! ** [ 0.184009] ** ** [ 0.185009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186008] ************************************************************* [ 0.187674] NET: Registered protocol family 16 [ 0.189486] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198084] cpuidle: using governor menu [ 0.201248] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.203663] PCI: Using configuration type 1 for base access [ 0.206123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.216184] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.217015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220603] cryptd: max_cpu_qlen set to 1000 [ 0.221204] ACPI: Added _OSI(Module Device) [ 0.222010] ACPI: Added _OSI(Processor Device) [ 0.223010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.225010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.231099] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237919] ACPI: Interpreter enabled [ 0.239048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.241013] ACPI: Using IOAPIC for interrupt routing [ 0.242244] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.246510] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.259050] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.271796] acpiphp: Slot [2] registered [ 0.273136] acpiphp: Slot [5] registered [ 0.275116] acpiphp: Slot [6] registered [ 0.277098] acpiphp: Slot [7] registered [ 0.278086] acpiphp: Slot [8] registered [ 0.279147] acpiphp: Slot [9] registered [ 0.281159] acpiphp: Slot [10] registered [ 0.282237] acpiphp: Slot [3] registered [ 0.284149] acpiphp: Slot [4] registered [ 0.285073] acpiphp: Slot [11] registered [ 0.287075] acpiphp: Slot [12] registered [ 0.288089] acpiphp: Slot [13] registered [ 0.289209] acpiphp: Slot [14] registered [ 0.291064] acpiphp: Slot [15] registered [ 0.292063] acpiphp: Slot [16] registered [ 0.294077] acpiphp: Slot [17] registered [ 0.295092] acpiphp: Slot [18] registered [ 0.297065] acpiphp: Slot [19] registered [ 0.298099] acpiphp: Slot [20] registered [ 0.300080] acpiphp: Slot [21] registered [ 0.301163] acpiphp: Slot [22] registered [ 0.303211] acpiphp: Slot [23] registered [ 0.305080] acpiphp: Slot [24] registered [ 0.306086] acpiphp: Slot [25] registered [ 0.307125] acpiphp: Slot [26] registered [ 0.309080] acpiphp: Slot [27] registered [ 0.310122] acpiphp: Slot [28] registered [ 0.312075] acpiphp: Slot [29] registered [ 0.314097] acpiphp: Slot [30] registered [ 0.316091] acpiphp: Slot [31] registered [ 0.318050] PCI host bridge to bus 0000:00 [ 0.319015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.328017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.331099] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333147] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.336067] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339268] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.352015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.357179] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.360011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.363022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.367017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.370526] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373781] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.376033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.378476] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.385013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.398013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.406012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.412615] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.438016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.460018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.494014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.508217] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.531014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.551017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.569016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.580606] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.592017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.601014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.628017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.645646] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.666016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.683012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.702016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.718626] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.739020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.756016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.783016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.801874] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.817015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.835020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.875017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.896780] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.900359] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.903622] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.907449] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.910684] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.916097] iommu: Default domain type: Passthrough [ 0.917516] SCSI subsystem initialized [ 0.919121] ACPI: bus type USB registered [ 0.921104] usbcore: registered new interface driver usbfs [ 0.923057] usbcore: registered new interface driver hub [ 0.925051] usbcore: registered new device driver usb [ 0.927126] pps_core: LinuxPPS API ver. 1 registered [ 0.928008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.932201] PTP clock support registered [ 0.935061] EDAC MC: Ver: 3.0.0 [ 0.937656] PCI: Using ACPI for IRQ routing [ 0.940031] NetLabel: Initializing [ 0.941007] NetLabel: domain hash size = 128 [ 0.943010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.946069] NetLabel: unlabeled traffic allowed by default [ 0.948395] vgaarb: loaded [ 0.950548] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.953014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.960169] clocksource: Switched to clocksource kvm-clock [ 1.135688] VFS: Disk quotas dquot_6.6.0 [ 1.137808] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.141346] *** VALIDATE ramfs *** [ 1.142769] *** VALIDATE hugetlbfs *** [ 1.145321] pnp: PnP ACPI init [ 1.148905] pnp: PnP ACPI: found 6 devices [ 1.169900] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.172735] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.174537] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.176334] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.178124] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.180153] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.182340] NET: Registered protocol family 2 [ 1.184363] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.188838] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.191914] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.196779] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.200908] TCP: Hash tables configured (established 65536 bind 65536) [ 1.204820] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.209904] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.214374] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.218445] NET: Registered protocol family 1 [ 1.222641] RPC: Registered named UNIX socket transport module. [ 1.227531] RPC: Registered udp transport module. [ 1.229166] RPC: Registered tcp transport module. [ 1.231622] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.234851] NET: Registered protocol family 44 [ 1.237035] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.239839] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.242535] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.245353] PCI: CLS 0 bytes, default 64 [ 1.247477] Unpacking initramfs... [ 3.043547] debug: unmapping init [mem 0xffff8b2cfcc54000-0xffff8b2cfffbffff] [ 3.047682] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.049670] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.052363] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.967920] Initialise system trusted keyrings [ 3.974311] Key type blacklist registered [ 3.977972] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.000825] zbud: loaded [ 4.008854] *** VALIDATE nfs *** [ 4.011985] *** VALIDATE nfs4 *** [ 4.018408] pstore: using deflate compression [ 4.027716] Platform Keyring initialized [ 4.251092] NET: Registered protocol family 38 [ 4.280614] Key type asymmetric registered [ 4.282296] Asymmetric key parser 'x509' registered [ 4.285112] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.288303] io scheduler mq-deadline registered [ 4.289861] io scheduler kyber registered [ 4.291940] io scheduler bfq registered [ 4.293809] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.297081] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.299838] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.302867] ACPI: Power Button [PWRF] [ 4.316652] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.331751] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.363697] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.383703] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.408403] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.446356] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.485310] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.491378] Non-volatile memory driver v1.3 [ 4.493080] Linux agpgart interface v0.103 [ 4.523764] virtio_blk virtio1: [vda] 134592 512-byte logical blocks (68.9 MB/65.7 MiB) [ 4.526564] vda: detected capacity change from 0 to 68911104 [ 4.564317] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.567752] vdb: detected capacity change from 0 to 1073741824 [ 4.632438] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.636232] vdc: detected capacity change from 0 to 2621440000 [ 4.656478] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.659653] vdd: detected capacity change from 0 to 2621440000 [ 4.675890] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.678354] vde: detected capacity change from 0 to 4294967296 [ 4.693722] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.696348] vdf: detected capacity change from 0 to 4294967296 [ 4.703474] libphy: Fixed MDIO Bus: probed [ 4.722502] usbcore: registered new interface driver usbserial_generic [ 4.724989] usbserial: USB Serial support registered for generic [ 4.729390] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.733358] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.734880] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.736922] mousedev: PS/2 mouse device common for all mice [ 4.741221] rtc_cmos 00:05: RTC can wake from S4 [ 4.742696] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.745759] rtc_cmos 00:05: registered as rtc0 [ 4.748425] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.749540] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.751807] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.753735] intel_pstate: CPU model not supported [ 4.758714] hid: raw HID events driver (C) Jiri Kosina [ 4.760337] usbcore: registered new interface driver usbhid [ 4.762111] usbhid: USB HID core driver [ 4.763396] drop_monitor: Initializing network drop monitor service [ 4.765284] Initializing XFRM netlink socket [ 4.766756] NET: Registered protocol family 10 [ 4.769342] Segment Routing with IPv6 [ 4.770486] NET: Registered protocol family 17 [ 4.771993] mpls_gso: MPLS GSO support [ 4.778490] RAS: Correctable Errors collector initialized. [ 4.780108] AVX version of gcm_enc/dec engaged. [ 4.781415] AES CTR mode by8 optimization enabled [ 4.909784] sched_clock: Marking stable (4909721579, 0)->(6482420271, -1572698692) [ 4.912981] registered taskstats version 1 [ 4.914623] Loading compiled-in X.509 certificates [ 4.916443] zswap: loaded using pool lzo/zbud [ 4.946473] Key type big_key registered [ 4.958306] Key type encrypted registered [ 4.959650] ima: No TPM chip found, activating TPM-bypass! [ 4.961553] ima: Allocated hash algorithm: sha1 [ 4.962897] ima: No architecture policies found [ 4.964647] evm: Initialising EVM extended attributes: [ 4.966286] evm: security.selinux [ 4.967357] evm: security.ima [ 4.968477] evm: security.capability [ 4.969652] evm: HMAC attrs: 0x1 [ 4.972230] rtc_cmos 00:05: setting system clock to 2026-01-10 01:21:41 UTC (1768008101) [ 4.978832] debug: unmapping init [mem 0xffffffff85203000-0xffffffff853fffff] [ 4.981518] debug: unmapping init [mem 0xffffffff83f82000-0xffffffff84258fff] [ 4.989156] Write protecting the kernel read-only data: 28672k [ 4.995747] debug: unmapping init [mem 0xffffffff82603000-0xffffffff827fffff] [ 4.998048] debug: unmapping init [mem 0xffffffff82f14000-0xffffffff82ffffff] [ 5.035783] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.043637] systemd[1]: Detected virtualization kvm. [ 5.045329] systemd[1]: Detected architecture x86-64. [ 5.046743] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.081532] systemd[1]: No hostname configured. [ 5.084188] systemd[1]: Set hostname to . [ 5.085748] random: systemd: uninitialized urandom read (16 bytes read) [ 5.087692] systemd[1]: Initializing machine ID from random generator. [ 5.210925] random: ln: uninitialized urandom read (6 bytes read) [ 5.380842] random: systemd: uninitialized urandom read (16 bytes read) [ 5.384452] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.394567] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.404568] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.512251] device-mapper: uevent: version 1.0.3 [ 6.514677] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.919321] random: fast init done [ 7.929662] virtio_net virtio0 ens2: renamed from eth0 [ 8.065075] scsi host0: ata_piix [ 8.238707] scsi host1: ata_piix [ 8.242253] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.246475] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.490733] random: crng init done [ 13.492022] random: 7 urandom warning(s) missed due to ratelimiting [ 16.438824] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.129898] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.798863] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.200957] SELinux: Disabled at runtime. [ 21.463924] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.683298] systemd[1]: Detected virtualization kvm. [ 21.686501] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.372992] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.377709] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.392619] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.402630] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.412306] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.424261] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.433213] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Created slice Us[ 23.654146] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS er and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 24.961359] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.092884] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.174394] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.571149] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.645142] EDAC sbridge: Ver: 1.1.2 [ 29.200938] Key type dns_resolver registered [ 29.693775] NFS: Registering the id_resolver key type [ 29.696260] Key type id_resolver registered [ 29.698425] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg309-server login: [ 58.922995] libcfs: loading out-of-tree module taints kernel. [ 58.942301] Key type ._llcrypt registered [ 58.946638] Key type .llcrypt registered [ 58.992833] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 67.528532] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 68.168856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.176560] alg: No test for adler32 (adler32-zlib) [ 69.194188] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 69.538822] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 71.159199] Key type lgssc registered [ 71.766879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.627623] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 80.386059] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.398187] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.678540] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.118231] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.423521] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 104.038332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 104.064928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 104.075560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 105.184163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 105.199445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 105.241899] Lustre: lustre-MDT0000: new disk, initializing [ 105.290793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 105.300361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 106.918470] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 112.815968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.844994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 112.901449] Lustre: 6485:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 112.925214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 112.928768] Lustre: Skipped 1 previous similar message [ 112.976898] Lustre: lustre-MDT0001: new disk, initializing [ 113.015629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 113.035082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 113.039607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 114.764411] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 117.297522] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 121.298353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 121.336729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 121.434480] Lustre: lustre-OST0000: new disk, initializing [ 121.436736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 121.463140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 123.901266] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 126.447704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 126.451064] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 126.466091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 129.822924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 129.845988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 129.881663] Lustre: lustre-OST0001: new disk, initializing [ 129.883544] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 129.900781] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 131.819248] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 134.637590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 134.640484] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 134.652587] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 140.272772] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 145.915297] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 155.809512] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing check_logdir /tmp/testlogs/ [ 159.207477] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing yml_node [ 162.698777] Lustre: DEBUG MARKER: Client: 2.17.0.RC3 [ 164.641974] Lustre: DEBUG MARKER: MDS: 2.17.0.RC3 [ 166.803663] Lustre: DEBUG MARKER: OSS: 2.17.0.RC3 [ 167.930417] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jan 9 20:24:23 EST 2026 [ 177.892369] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 178.575278] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 195.041268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 195.050211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 195.061879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 196.064395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 196.064642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 196.071800] Lustre: Skipped 1 previous similar message [ 196.079293] Lustre: Skipped 1 previous similar message [ 197.202566] LustreError: 12188:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 197.262835] Lustre: server umount lustre-MDT0000 complete [ 199.398699] LustreError: 8394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768008295 with bad export cookie 3062619272718522957 [ 199.400683] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 199.411627] LustreError: 8394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 201.184365] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 201.185467] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 201.194803] Lustre: Skipped 2 previous similar messages [ 201.204434] Lustre: Skipped 1 previous similar message [ 204.192176] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 204.199983] Lustre: Skipped 1 previous similar message [ 205.574678] LustreError: 12390:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 205.577121] LustreError: 12390:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 205.646161] Lustre: server umount lustre-MDT0001 complete [ 207.782887] LustreError: 12592:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 207.784934] LustreError: 12592:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 207.806227] Lustre: server umount lustre-OST0000 complete [ 209.529324] Lustre: server umount lustre-OST0001 complete [ 212.621226] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 216.964442] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 222.248082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 225.780564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 229.280424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 232.948414] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 236.969895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 237.002753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 237.126348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 237.143701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 237.193927] Lustre: lustre-MDT0000: new disk, initializing [ 237.244360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.253870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 239.409939] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 245.551096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 245.597241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 245.642247] Lustre: 14974:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 245.672531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 245.675564] Lustre: Skipped 1 previous similar message [ 245.729801] Lustre: lustre-MDT0001: new disk, initializing [ 245.788769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 245.808776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 245.817720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 247.865170] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 252.209681] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 254.133759] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 257.486610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 257.510145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 259.536751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 259.588288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 260.400451] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 264.285525] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 266.152487] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 266.249591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 268.180086] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 268.284716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 269.795351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 269.795446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.798962] LustreError: Skipped 1 previous similar message [ 269.800766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 269.809647] Lustre: Skipped 1 previous similar message [ 275.256082] LustreError: 17910:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 275.259274] LustreError: 17910:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 275.339383] Lustre: server umount lustre-OST0000 complete [ 280.033923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.036642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 280.045380] Lustre: Skipped 1 previous similar message [ 280.049014] Lustre: Skipped 3 previous similar messages [ 283.421315] LustreError: 18112:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 283.428298] LustreError: 18112:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 283.492135] Lustre: server umount lustre-MDT0000 complete [ 285.154261] LustreError: 14985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 285.176109] LustreError: 14985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 285.591153] LustreError: 18114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768008382 with bad export cookie 3062619272718524476 [ 285.593800] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 285.596389] LustreError: 18114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 285.775548] Lustre: server umount lustre-MDT0001 complete [ 288.962149] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 20:26:24 (1768008384) [ 298.986955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 299.259881] LustreError: 19449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.297322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 299.300637] Lustre: Skipped 1 previous similar message [ 301.483917] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 304.609510] LustreError: 19449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 306.587590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 309.015816] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 312.262412] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 314.335996] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 315.541391] Lustre: Failing over lustre-MDT0000 [ 315.552361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.554990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.572891] Lustre: Skipped 1 previous similar message [ 315.583060] Lustre: Skipped 2 previous similar messages [ 315.673634] LustreError: 20791:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 315.682106] LustreError: 20791:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 315.733290] Lustre: server umount lustre-MDT0000 complete [ 316.645841] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.663966] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 322.017346] LustreError: 19473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.031448] LustreError: 19473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 330.541452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 330.621114] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 331.754418] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 331.757061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 331.767942] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 332.678456] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 337.065443] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 341.988405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.988855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 345.622930] LustreError: 21771:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 345.629479] LustreError: 21771:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 345.698371] Lustre: server umount lustre-MDT0000 complete [ 347.104631] LustreError: 19444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 347.114658] LustreError: 19444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 347.916445] LustreError: 19429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768008444 with bad export cookie 3062619272718525890 [ 347.919536] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 347.927078] LustreError: 19429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 348.083841] Lustre: server umount lustre-MDT0001 complete [ 352.800885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 353.118605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 353.124226] Lustre: Skipped 2 previous similar messages [ 355.383519] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 360.813354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 364.058183] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 367.288701] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 369.835201] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 371.315142] Lustre: Failing over lustre-MDT0000 [ 371.513086] Lustre: server umount lustre-MDT0000 complete [ 371.680367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 371.686824] LustreError: 22449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.703819] LustreError: 22449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 376.800962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.810236] Lustre: Skipped 2 previous similar messages [ 387.212677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.281830] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 389.524390] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 392.676908] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 392.679624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 392.694781] Lustre: Skipped 1 previous similar message [ 392.709330] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 394.259617] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 397.792546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 397.800921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 397.803605] Lustre: Skipped 2 previous similar messages [ 403.349987] LustreError: 24769:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 403.353507] LustreError: 24769:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 403.435482] Lustre: server umount lustre-MDT0000 complete [ 405.811444] LustreError: 22431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768008502 with bad export cookie 3062619272718526821 [ 405.814460] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 405.819440] LustreError: 22431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 406.080586] Lustre: server umount lustre-MDT0001 complete [ 410.959398] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 20:28:26 (1768008506) [ 419.310421] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 424.489265] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 429.483220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 433.185655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 436.377840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 439.541781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 444.450277] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 449.303901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 449.335771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.458398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 449.479346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 449.481981] Lustre: Skipped 1 previous similar message [ 449.526616] Lustre: lustre-MDT0000: new disk, initializing [ 449.527940] Lustre: Skipped 1 previous similar message [ 449.571520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.574116] Lustre: Skipped 2 previous similar messages [ 449.584017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 449.588500] Lustre: Skipped 1 previous similar message [ 451.264682] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 456.825251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 456.862427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 456.916867] Lustre: 28432:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 457.067111] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 459.179262] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 463.463358] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 465.058487] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 468.143812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 468.178364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 469.902869] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 469.923971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 469.930978] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 470.786469] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 474.756730] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 478.733609] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 490.469650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 490.469701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.475274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 490.475281] Lustre: Skipped 2 previous similar messages [ 490.477566] LustreError: Skipped 1 previous similar message [ 490.489610] Lustre: Skipped 3 previous similar messages [ 492.694256] LustreError: 31070:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 492.697744] LustreError: 31070:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 492.785061] Lustre: server umount lustre-OST0000 complete [ 501.036484] Lustre: server umount lustre-MDT0000 complete [ 503.011851] LustreError: 28425:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768008599 with bad export cookie 3062619272718527220 [ 503.017615] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 503.024295] LustreError: 28425:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 503.294951] Lustre: server umount lustre-MDT0001 complete [ 508.581111] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 509.982088] Key type lgssc unregistered [ 510.186756] LNet: 32003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 510.192680] LNetError: 32003:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 510.204719] LNet: Removed LNI 192.168.203.109@tcp [ 510.709157] Key type .llcrypt unregistered [ 510.710849] Key type ._llcrypt unregistered [ 522.861048] Key type ._llcrypt registered [ 522.862681] Key type .llcrypt registered [ 522.911361] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 523.313189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 523.366203] alg: No test for adler32 (adler32-zlib) [ 524.257262] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 524.386458] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 526.015166] Key type lgssc registered [ 526.506624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 530.007303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 530.020903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.323139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 533.093093] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 537.645702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 537.843724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 539.777920] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 542.474244] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 544.316590] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 547.834383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 548.013893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 549.051509] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 550.935927] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 553.715861] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 566.480257] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 20:31:02 (1768008662) [ 567.658298] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 568.625140] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 572.323296] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 20:31:08 (1768008668) [ 576.998674] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 20:31:12 (1768008672) [ 577.486460] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 582.174810] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 582.178223] Lustre: Skipped 2 previous similar messages [ 583.130802] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 20:31:18 (1768008678) [ 587.087049] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 587.090841] Lustre: Skipped 4 previous similar messages [ 594.054479] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 20:31:30 (1768008690) [ 595.268609] Lustre: *** cfs_fail_loc=131b, val=2*** [ 596.701637] Lustre: *** cfs_fail_loc=131b, val=2*** [ 600.181620] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 20:31:36 (1768008696) [ 601.015941] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 601.019461] Lustre: Skipped 9 previous similar messages [ 606.739688] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 20:31:42 (1768008702) [ 612.833908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 612.834137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 612.836115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 612.839952] Lustre: Skipped 1 previous similar message [ 617.953255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 617.956372] Lustre: Skipped 2 previous similar messages [ 618.094127] LustreError: 39925:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 618.138244] Lustre: server umount lustre-OST0000 complete [ 619.999784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 620.003048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.008754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.072211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.073140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.797935] LustreError: 40126:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 625.800172] LustreError: 40126:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 625.830216] Lustre: server umount lustre-MDT0000 complete [ 627.433455] LustreError: 34992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768008723 with bad export cookie 2131467594343930251 [ 627.437231] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 627.440371] LustreError: 34992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 627.527851] LustreError: 40327:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 627.529608] LustreError: 40327:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 627.626854] Lustre: server umount lustre-MDT0001 complete [ 632.427479] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 633.627954] Key type lgssc unregistered [ 633.773703] LNet: 40858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.776882] LNetError: 40858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.785546] LNet: Removed LNI 192.168.203.109@tcp [ 634.178951] Key type .llcrypt unregistered [ 634.180232] Key type ._llcrypt unregistered [ 640.769988] Key type ._llcrypt registered [ 640.771283] Key type .llcrypt registered [ 640.812559] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 647.705807] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 648.056370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 648.128049] alg: No test for adler32 (adler32-zlib) [ 649.005054] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 649.107001] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 650.703162] Key type lgssc registered [ 651.131853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 654.596129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 657.285362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 659.791235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 662.389318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 665.183887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 665.208957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 665.214835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 666.321704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 666.348224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 666.396442] Lustre: lustre-MDT0000: new disk, initializing [ 666.439379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 666.448150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 668.033861] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 672.834839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 672.862103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 672.912081] Lustre: 43963:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 672.940204] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 672.943630] Lustre: Skipped 1 previous similar message [ 672.983859] Lustre: lustre-MDT0001: new disk, initializing [ 673.020819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 673.035820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 673.040699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 674.632304] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 678.106253] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 679.686198] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 681.584415] hrtimer: interrupt took 3048286 ns [ 682.774567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 682.815822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 682.956089] Lustre: lustre-OST0000: new disk, initializing [ 682.958152] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 682.991845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 685.234684] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 688.687238] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 690.339585] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 692.730070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 692.734598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 692.767683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 693.497256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 694.912646] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 694.995582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 697.828228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 697.828635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.832270] LustreError: Skipped 1 previous similar message [ 697.837881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 697.841754] Lustre: Skipped 1 previous similar message [ 701.751088] LustreError: 46924:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 701.796404] Lustre: server umount lustre-OST0000 complete [ 703.967828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 703.971208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.976194] Lustre: Skipped 1 previous similar message [ 703.978803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.064224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.064926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.377693] LustreError: 47126:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 709.380846] LustreError: 47126:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 709.415860] Lustre: server umount lustre-MDT0000 complete [ 710.837317] LustreError: 43953:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768008807 with bad export cookie 11172269351640722370 [ 710.840463] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 710.841486] LustreError: 43953:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 710.914036] LustreError: 47327:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 710.916980] LustreError: 47327:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 710.973276] Lustre: server umount lustre-MDT0001 complete [ 714.674677] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 20:33:30 (1768008810) [ 717.324143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 717.537512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 718.991738] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 722.214832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 722.354098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 723.792515] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 725.642677] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 726.984826] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 729.596380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 731.810479] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 733.954585] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 746.327836] LustreError: 50263:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 746.694049] Lustre: Modifying parameter general.timeout=1 in log params [ 747.754041] Lustre: Modifying parameter general.timeout=4 in log params [ 747.755870] Lustre: Skipped 2 previous similar messages [ 749.791838] Lustre: Modifying parameter general.timeout=10 in log params [ 749.794521] Lustre: Skipped 5 previous similar messages [ 753.946606] Lustre: Modifying parameter general.timeout=22 in log params [ 753.948407] Lustre: Skipped 11 previous similar messages [ 762.032679] Lustre: Modifying parameter general.timeout=46 in log params [ 762.035294] Lustre: Skipped 23 previous similar messages [ 778.069816] Lustre: Modifying parameter general.timeout=90 in log params [ 778.073169] Lustre: Skipped 43 previous similar messages [ 796.206592] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 20:34:52 (1768008892) [ 797.411741] Lustre: 57165:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 798.889564] Lustre: 57361:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 798.895898] Lustre: 57361:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 801.174962] Lustre: 57655:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 801.180052] Lustre: 57655:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 811.729047] Lustre: 59002:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 811.736865] LustreError: 59002:0:(mgs_handler.c:1129:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 812.085455] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_policies= in log params [ 812.088142] Lustre: Skipped 60 previous similar messages [ 813.304260] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 20:35:09 (1768008909) [ 819.679825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 819.684093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.691385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 819.694838] Lustre: Skipped 1 previous similar message [ 823.679095] LustreError: 59698:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 823.681589] LustreError: 59698:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 823.707097] Lustre: server umount lustre-MDT0000 complete [ 825.173225] LustreError: 48024:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768008921 with bad export cookie 11172269351640723238 [ 825.175925] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 825.178626] LustreError: 48024:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 825.306610] Lustre: server umount lustre-MDT0001 complete [ 837.105456] LustreError: 60103:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 837.108251] LustreError: 60103:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 837.157270] Lustre: server umount lustre-OST0000 complete [ 840.647434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 842.141236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 843.633682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 845.120265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 847.867551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.999395] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 848.106163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.109044] Lustre: Skipped 1 previous similar message [ 849.678701] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 853.087788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 853.153905] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 853.168952] Lustre: 61552:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 854.940692] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 857.201661] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 858.604051] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 861.527809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 861.643174] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 863.996857] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 866.113715] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 872.928675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 872.933822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.940675] Lustre: Skipped 2 previous similar messages [ 872.943248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 872.946387] Lustre: Skipped 3 previous similar messages [ 875.061252] LustreError: 64034:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 875.063906] LustreError: 64034:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 875.108593] Lustre: server umount lustre-OST0000 complete [ 883.680139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 883.684657] Lustre: Skipped 5 previous similar messages [ 890.847115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 890.913763] Lustre: server umount lustre-MDT0000 complete [ 892.339310] LustreError: 61544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768008988 with bad export cookie 11172269351640725660 [ 892.341898] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 892.343539] LustreError: 61544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 892.411970] LustreError: 64444:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 892.415674] LustreError: 64444:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 892.469508] Lustre: server umount lustre-MDT0001 complete [ 896.296033] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 897.408587] Key type lgssc unregistered [ 897.541309] LNet: 64975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.544553] LNetError: 64975:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.554337] LNet: Removed LNI 192.168.203.109@tcp [ 897.862089] Key type .llcrypt unregistered [ 897.863906] Key type ._llcrypt unregistered [ 907.738775] Key type ._llcrypt registered [ 907.740230] Key type .llcrypt registered [ 907.790584] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 908.177608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 908.215303] alg: No test for adler32 (adler32-zlib) [ 909.075891] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 909.163583] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 910.751185] Key type lgssc registered [ 911.188099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 914.177390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 914.184766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 915.374108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.883176] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 920.274917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 920.511077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 922.053744] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 924.104456] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 925.427345] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 928.134212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 928.400834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 930.689162] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 932.969328] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 946.412465] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 950.241263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 950.242754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 950.246505] Lustre: Skipped 2 previous similar messages [ 950.249296] Lustre: Skipped 1 previous similar message [ 955.262924] LustreError: 69071:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 955.290359] Lustre: server umount lustre-MDT0000 complete [ 955.364082] LustreError: 66348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 955.371098] LustreError: 66348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 956.746388] LustreError: 68717:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768009053 with bad export cookie 17587886840805425335 [ 956.748737] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 956.750826] LustreError: 68717:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 956.828711] LustreError: 69276:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 956.831372] LustreError: 69276:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 956.889664] Lustre: server umount lustre-MDT0001 complete [ 967.604243] LustreError: 69477:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 967.606830] LustreError: 69477:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 967.661911] Lustre: server umount lustre-OST0000 complete [ 971.031688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 972.552660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 974.093052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 975.583034] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 978.215859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.320263] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 978.332960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 978.442563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 979.919506] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 983.225221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 983.272813] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 983.284799] Lustre: 70924:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 983.473433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 985.089585] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 987.270506] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 988.695926] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 991.357591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 991.432728] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 991.586834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.705285] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 995.736535] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 996.318341] Lustre: 73167:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 997.013312] Lustre: Modifying parameter general.jobid_var=disable in log params [ 998.881615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 998.882581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.885071] LustreError: Skipped 1 previous similar message [ 998.885770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 998.890033] Lustre: Skipped 1 previous similar message [ 1000.430134] LustreError: 73415:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1000.432391] LustreError: 73415:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1000.472152] Lustre: server umount lustre-OST0000 complete [ 1003.935227] Lustre: server umount lustre-MDT0000 complete [ 1004.003067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.006102] LustreError: 70932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.012607] Lustre: Skipped 1 previous similar message [ 1004.021649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1005.445197] LustreError: 70918:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768009101 with bad export cookie 17587886840805426147 [ 1005.447621] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1005.449033] LustreError: 70918:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1005.523275] LustreError: 73823:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1005.527287] LustreError: 73823:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 1005.584764] Lustre: server umount lustre-MDT0001 complete [ 1009.678286] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1010.746700] Key type lgssc unregistered [ 1010.869856] LNet: 74354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.873365] LNetError: 74354:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.887444] LNet: Removed LNI 192.168.203.109@tcp [ 1011.204566] Key type .llcrypt unregistered [ 1011.206512] Key type ._llcrypt unregistered [ 1021.310072] Key type ._llcrypt registered [ 1021.311748] Key type .llcrypt registered [ 1021.356363] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1021.647207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1021.657829] alg: No test for adler32 (adler32-zlib) [ 1022.526867] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 1022.615140] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1024.199153] Key type lgssc registered [ 1024.608254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1027.627201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1027.641904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.881767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1030.397228] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1033.600279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1033.815725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1035.243922] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1037.150709] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1038.379575] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1040.700320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1040.887977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1042.828664] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1044.789644] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1062.514837] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1063.139761] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1063.141859] Lustre: Skipped 1 previous similar message [ 1064.440730] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1064.443273] Lustre: Skipped 3 previous similar messages [ 1066.756981] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1066.760011] Lustre: Skipped 6 previous similar messages [ 1070.971156] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1070.973808] Lustre: Skipped 11 previous similar messages [ 1079.067612] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1079.070443] Lustre: Skipped 23 previous similar messages [ 1095.175349] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1095.178089] Lustre: Skipped 48 previous similar messages [ 1098.486295] Lustre: 83485:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1099.389272] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 20:39:55 (1768009195) [ 1099.945678] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1100.623194] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 20:39:56 (1768009196) [ 1105.376296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1105.379958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.386250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1106.401109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1106.401758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1106.401938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.411753] Lustre: Skipped 1 previous similar message [ 1111.522128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1111.524296] Lustre: Skipped 1 previous similar message [ 1116.127109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1116.146099] LustreError: 83776:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1116.183736] Lustre: server umount lustre-OST0000 complete [ 1120.735853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1120.738311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.742671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1123.647205] LustreError: 83982:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1123.650738] LustreError: 83982:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1123.683125] Lustre: server umount lustre-MDT0000 complete [ 1125.027402] LustreError: 75706:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768009221 with bad export cookie 12398248374586478430 [ 1125.027965] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1125.032258] LustreError: 75706:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1125.100665] LustreError: 84183:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1125.103471] LustreError: 84183:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1125.167593] Lustre: server umount lustre-MDT0001 complete [ 1128.908208] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1129.945842] Key type lgssc unregistered [ 1130.067387] LNet: 84713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1130.069772] LNetError: 84713:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1130.078337] LNet: Removed LNI 192.168.203.109@tcp [ 1130.372500] Key type .llcrypt unregistered [ 1130.373760] Key type ._llcrypt unregistered [ 1131.729767] Key type ._llcrypt registered [ 1131.731116] Key type .llcrypt registered [ 1131.768626] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1133.190953] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules [ 1133.512339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1133.565558] alg: No test for adler32 (adler32-zlib) [ 1134.421375] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 1134.527590] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1136.111140] Key type lgssc registered [ 1136.519095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1136.917085] LustreError: 85830:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1144.676193] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1148.092905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1158.149621] LustreError: 85830:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1158.161619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.350456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1160.750044] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1163.934447] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 20:40:59 (1768009259) [ 1165.625956] LustreError: 86867:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1165.629665] LustreError: 86867:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1165.632391] LustreError: 86867:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1165.701957] LustreError: 87639:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1165.766030] Lustre: server umount lustre-MDT0000 complete [ 1170.447801] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1171.473569] Key type lgssc unregistered [ 1171.604428] LNet: 88324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.607909] LNetError: 88324:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.615291] LNet: Removed LNI 192.168.203.109@tcp [ 1171.937217] Key type .llcrypt unregistered [ 1171.938330] Key type ._llcrypt unregistered [ 1179.159484] Key type ._llcrypt registered [ 1179.160811] Key type .llcrypt registered [ 1179.205331] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1179.532666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1179.563080] alg: No test for adler32 (adler32-zlib) [ 1180.428238] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 1180.520611] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1182.103150] Key type lgssc registered [ 1182.503899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1185.639338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1185.648545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1186.871748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1188.299671] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1191.481560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1191.752445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1193.197473] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1195.193766] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1196.536812] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1198.866146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1199.093431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1201.100705] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1202.972345] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1234.440890] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 20:42:10 (1768009330) [ 1234.986754] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1235.629040] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 20:42:11 (1768009331) [ 1237.985234] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.986521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1237.986668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1237.989867] Lustre: Skipped 1 previous similar message [ 1242.749602] LustreError: 92430:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1242.778742] Lustre: server umount lustre-MDT0000 complete [ 1243.105265] LustreError: 91583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1243.110665] LustreError: 91583:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1243.940075] LustreError: 89482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768009340 with bad export cookie 18129204028645330880 [ 1243.942877] LustreError: 89482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1243.942904] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1244.004343] LustreError: 92633:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1244.006358] LustreError: 92633:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1244.049628] Lustre: server umount lustre-MDT0001 complete [ 1255.414551] LustreError: 92833:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1255.416696] LustreError: 92833:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1255.538155] Lustre: server umount lustre-OST0000 complete [ 1259.483067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.641834] LustreError: 93462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1259.703962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1261.050205] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1263.859555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1264.099800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1265.523678] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1267.261307] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1268.391337] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1270.324719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1272.542965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1272.563815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1272.639683] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1272.642079] LustreError: 93450:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1272.651086] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1272.653974] LustreError: 95324:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1272.656450] LustreError: 95324:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1272.658425] LustreError: 95324:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1272.682286] Lustre: server umount lustre-OST0000 complete [ 1272.683913] LustreError: 95324:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1275.609307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1275.673080] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1275.676696] LustreError: 93451:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1275.690912] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1275.694959] LustreError: 95790:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1275.698392] LustreError: 95790:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1275.701528] LustreError: 95790:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1275.730407] Lustre: server umount lustre-OST0000 complete [ 1275.732166] LustreError: 95790:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1276.728848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1279.000709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1279.084652] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1279.086160] Lustre: Found index 0 for lustre-OST0000, updating log [ 1279.088184] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1279.112163] Lustre: lustre-OST0000: new disk, initializing [ 1279.114739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1279.212519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.260772] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1282.678769] Lustre: Failing over lustre-OST0000 [ 1282.700172] LustreError: 96865:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1282.702353] LustreError: 96865:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1282.738945] Lustre: server umount lustre-OST0000 complete [ 1284.084259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.084902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1284.088525] Lustre: Skipped 2 previous similar messages [ 1284.090680] Lustre: Skipped 2 previous similar messages [ 1289.183975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1289.186112] Lustre: Skipped 2 previous similar messages [ 1295.327980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1295.330162] Lustre: Skipped 3 previous similar messages [ 1298.399134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1298.440627] LustreError: 97065:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1298.443547] LustreError: 97065:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1298.470163] Lustre: server umount lustre-MDT0000 complete [ 1299.948250] LustreError: 94187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768009396 with bad export cookie 18129204028645333610 [ 1299.950603] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1299.952026] LustreError: 94187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1300.098978] Lustre: server umount lustre-MDT0001 complete [ 1303.350754] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 20:43:19 (1768009399) [ 1308.568959] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1312.302430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1312.480165] LustreError: 98637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1312.511037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1313.907180] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1316.911776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1318.674289] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1319.782989] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1322.358688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1324.713653] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1327.885330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1327.911995] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1327.942547] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1327.980379] Lustre: lustre-OST0001: new disk, initializing [ 1327.982246] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1330.122315] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1333.425620] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1335.175066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1335.178591] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1335.203506] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1340.315161] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1348.064108] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1348.067484] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1348.073703] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1350.627718] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.633224] Lustre: Skipped 1 previous similar message [ 1356.717032] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.9@tcp (stopping) [ 1356.720057] Lustre: Skipped 7 previous similar messages [ 1361.375101] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1361.417179] LustreError: 101871:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1361.420349] LustreError: 101871:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1361.445720] Lustre: server umount lustre-MDT0001 complete [ 1361.837785] LustreError: 98631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.549107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1365.120967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1365.156472] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1365.159385] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1365.182797] Lustre: 98624:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1365.461110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1365.464208] Lustre: Skipped 3 previous similar messages [ 1366.629386] LustreError: 102585:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1366.632328] LustreError: 102585:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1366.635301] LustreError: 98628:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1366.662653] LustreError: 102594:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.109@tcp: cfg command failed: rc = -17 [ 1366.666362] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1366.666362] [ 1366.913333] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1370.148447] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 20:44:26 (1768009466) [ 1370.600440] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1376.076356] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1382.488869] Lustre: 104478:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1390.674591] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1391.808664] Lustre: 105609:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1442.272297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1442.272544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.273507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1442.286059] Lustre: Skipped 6 previous similar messages [ 1447.817067] LustreError: 106040:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1447.820592] LustreError: 106040:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1447.853330] Lustre: server umount lustre-MDT0000 complete [ 1449.185691] LustreError: 99746:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768009545 with bad export cookie 18129204028645334478 [ 1449.191496] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1449.321743] Lustre: server umount lustre-MDT0001 complete [ 1460.744319] Lustre: server umount lustre-OST0000 complete [ 1472.284266] Lustre: server umount lustre-OST0001 complete [ 1474.249086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1477.389425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1477.740052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1481.213436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1484.613849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1484.963771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1492.205838] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1496.002076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1496.013546] Lustre: lustre-MDT0000: reset Object Index mappings [ 1496.188623] LustreError: 109524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1496.227323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.680695] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1500.631160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1500.637099] Lustre: lustre-MDT0001: reset Object Index mappings [ 1502.278155] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1503.346530] Lustre: 110638:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1505.861407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1506.018339] LustreError: 110996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.697068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 1508.266817] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1510.828316] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 1511.435058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1512.881026] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 1512.927528] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 1513.957287] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1517.593382] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1518.971969] Lustre: 112493:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1539.739505] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 20:47:15 (1768009635) [ 1572.320285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1572.322771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1572.328020] Lustre: Skipped 3 previous similar messages [ 1572.329768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1572.332401] Lustre: Skipped 4 previous similar messages [ 1575.253474] LustreError: 112843:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1575.255501] LustreError: 112843:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1575.288582] Lustre: server umount lustre-MDT0000 complete [ 1576.733307] LustreError: 109503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768009673 with bad export cookie 18129204028645381175 [ 1576.736213] LustreError: 109503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1576.739095] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1578.464648] LustreError: 111814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1578.471499] LustreError: 111814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1591.263115] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1591.391226] Lustre: server umount lustre-MDT0001 complete [ 1607.135092] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1607.180553] Lustre: server umount lustre-OST0000 complete [ 1614.690862] Lustre: server umount lustre-OST0001 complete [ 1617.245572] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 1620.746914] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1624.682689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1627.347903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1629.895297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1632.577642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1635.505597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1635.537821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1635.641951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1635.655908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1635.695808] Lustre: lustre-MDT0000: new disk, initializing [ 1635.726705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1635.729307] Lustre: Skipped 3 previous similar messages [ 1635.736353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1637.385758] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1640.635521] LustreError: 116199:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1640.639377] LustreError: 116199:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1640.701916] Lustre: server umount lustre-MDT0000 complete [ 1641.800883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1644.552434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1644.584533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.752899] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1644.758051] Lustre: 116718:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1644.765608] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1646.456639] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1648.480295] Lustre: server umount lustre-MDT0000 complete [ 1651.784662] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 20:49:07 (1768009747) [ 1652.410788] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1653.256508] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 20:49:09 (1768009749) [ 1660.026667] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1664.204465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1666.070983] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1669.451943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1669.477919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1669.518297] Lustre: 118735:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1669.549037] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1669.551226] Lustre: Skipped 1 previous similar message [ 1669.589858] Lustre: lustre-MDT0001: new disk, initializing [ 1669.651649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1669.656663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1671.118959] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1673.582049] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1676.342291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1676.374890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1676.482518] Lustre: lustre-OST0000: new disk, initializing [ 1676.484992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1676.848501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1676.853367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1676.902378] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1678.727595] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1683.730709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1683.767647] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1685.268250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1685.272142] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1685.322381] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1686.375299] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1691.429183] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1692.896897] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1825.500702] Lustre: 127495:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1825.504409] Lustre: 127495:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1825.507375] Lustre: 127495:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1825.509924] Lustre: 127495:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1825.512375] Lustre: 127495:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1825.515054] Lustre: 127495:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1920.654427] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:53:36 (1768010016) [ 1935.839924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1935.842356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.845744] Lustre: Skipped 5 previous similar messages [ 1935.847463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1935.849821] Lustre: Skipped 10 previous similar messages [ 1938.306914] LustreError: 127703:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1938.310082] LustreError: 127703:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1938.374182] Lustre: server umount lustre-MDT0000 complete [ 1939.784227] LustreError: 119457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768010036 with bad export cookie 18129204028645434788 [ 1939.786389] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1939.788473] LustreError: 119457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1941.472612] LustreError: 121665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1941.478071] LustreError: 121665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1954.271124] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1954.390997] Lustre: server umount lustre-MDT0001 complete [ 1970.143107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1970.186717] Lustre: server umount lustre-OST0000 complete [ 1977.565813] Lustre: server umount lustre-OST0001 complete [ 1979.683437] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 1982.855865] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1986.465250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1988.804529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1990.858227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1992.965384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1995.192802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1995.212606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.296499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1995.306643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1995.308185] Lustre: Skipped 1 previous similar message [ 1995.344794] Lustre: lustre-MDT0000: new disk, initializing [ 1995.346200] Lustre: Skipped 1 previous similar message [ 1995.367399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.369375] Lustre: Skipped 5 previous similar messages [ 1995.375726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1996.682177] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2000.888429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2000.920726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2000.968616] Lustre: 130491:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2001.085958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2002.379576] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2005.543260] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2006.683595] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2008.982552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2009.004753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2010.684130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2010.728102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2011.197071] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2014.414480] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2017.771172] Lustre: lustre-MDD0000: changelog on [ 2018.727642] Lustre: lustre-MDD0001: changelog on [ 2378.655122] Lustre: 130496:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2406.478754] Lustre: lustre-MDD0001: changelog off [ 2407.682780] Lustre: lustre-MDD0000: changelog off [ 2409.954262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.954680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2409.955286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2409.955290] Lustre: Skipped 10 previous similar messages [ 2409.957907] Lustre: Skipped 6 previous similar messages [ 2415.406076] LustreError: 139975:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2415.409444] LustreError: 139975:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2415.455320] Lustre: server umount lustre-OST0000 complete [ 2422.913971] Lustre: server umount lustre-MDT0000 complete [ 2424.189224] LustreError: 131271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768010520 with bad export cookie 18129204028646236190 [ 2424.192074] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2424.196584] LustreError: 131271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2424.315253] Lustre: server umount lustre-MDT0001 complete [ 2428.014178] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2429.124447] Key type lgssc unregistered [ 2429.252444] LNet: 140908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2429.256809] LNetError: 140908:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2429.266453] LNet: Removed LNI 192.168.203.109@tcp [ 2429.614119] Key type .llcrypt unregistered [ 2429.615410] Key type ._llcrypt unregistered [ 2436.489665] Key type ._llcrypt registered [ 2436.490919] Key type .llcrypt registered [ 2436.531777] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 2442.810496] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2443.085633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2443.093380] alg: No test for adler32 (adler32-zlib) [ 2443.937922] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 2444.017346] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2445.591148] Key type lgssc registered [ 2445.915520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2448.839975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2450.898423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2452.774080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2454.741170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2456.961697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2456.977904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2456.981606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.073278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2458.087512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2458.138941] Lustre: lustre-MDT0000: new disk, initializing [ 2458.165979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2458.173243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2459.572341] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2463.639638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2463.663587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2463.704746] Lustre: 144057:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2463.734510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2463.736208] Lustre: Skipped 1 previous similar message [ 2463.770366] Lustre: lustre-MDT0001: new disk, initializing [ 2463.807700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2463.822065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2463.825680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2465.132842] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2468.156313] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2469.369514] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2471.617559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2471.644681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2471.780697] Lustre: lustre-OST0000: new disk, initializing [ 2471.784147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2471.818101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2473.740223] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2476.768773] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2480.629749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2480.633121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2480.660474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2484.079886] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_module obdecho/obdecho [ 2484.678177] LustreError: 146680:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2484.681127] LustreError: 146680:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 2485.256642] LustreError: 146776:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2489.311943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2489.314878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.320285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2490.849464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.849516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2490.854259] Lustre: Skipped 1 previous similar message [ 2490.855844] Lustre: Skipped 1 previous similar message [ 2494.335499] LustreError: 147116:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2494.363921] Lustre: server umount lustre-MDT0000 complete [ 2495.560150] LustreError: 144840:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768010592 with bad export cookie 526462594897197142 [ 2495.563384] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2495.564589] LustreError: 144840:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2495.625707] LustreError: 147317:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2495.627513] LustreError: 147317:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2495.679844] Lustre: server umount lustre-MDT0001 complete [ 2507.122619] LustreError: 147517:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2507.125190] LustreError: 147517:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2507.174682] Lustre: server umount lustre-OST0000 complete [ 2513.683203] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 2516.799300] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2519.982434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2521.944456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2523.810599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2525.792180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2527.960606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2527.986494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.061951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2528.070548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2528.101713] Lustre: lustre-MDT0000: new disk, initializing [ 2528.122340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.128279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2529.334896] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2533.291173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2533.313644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2533.351656] Lustre: 150457:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2533.372356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2533.374802] Lustre: Skipped 1 previous similar message [ 2533.396589] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2533.399024] Lustre: Skipped 1 previous similar message [ 2533.413293] Lustre: lustre-MDT0001: new disk, initializing [ 2533.444605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2533.456087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2533.459671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2534.648978] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2537.661054] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2538.756471] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2540.955246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2540.981921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2541.319225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2541.358401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2542.852473] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2545.747551] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2546.995874] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2547.057367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2548.249085] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2548.312837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2551.776600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2551.777297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2551.780322] LustreError: Skipped 1 previous similar message [ 2551.785349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2551.788860] Lustre: Skipped 1 previous similar message [ 2554.988130] LustreError: 153398:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2554.990958] LustreError: 153398:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2555.025895] Lustre: server umount lustre-OST0000 complete [ 2556.896175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.896657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.900576] Lustre: Skipped 2 previous similar messages [ 2562.301653] LustreError: 153599:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2562.303520] LustreError: 153599:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2562.330348] Lustre: server umount lustre-MDT0000 complete [ 2563.573192] LustreError: 151236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768010660 with bad export cookie 526462594897198073 [ 2563.576646] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2563.577039] LustreError: 151236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2563.677599] Lustre: server umount lustre-MDT0001 complete [ 2564.760056] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:04:20 (1768010660) [ 2565.228275] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2565.767235] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:04:21 (1768010661) [ 2568.169152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.330040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.333167] Lustre: Skipped 1 previous similar message [ 2569.574151] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2572.234240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2573.615736] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2575.331243] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2576.482699] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2578.704943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2580.641991] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2582.368259] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2602.933744] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.9@tcp (stopping) [ 2602.936885] Lustre: Skipped 3 previous similar messages [ 2602.975753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2602.978134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2605.758233] LustreError: 156739:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2605.760554] LustreError: 156739:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2605.791515] Lustre: server umount lustre-MDT0000 complete [ 2607.071213] LustreError: 156090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768010703 with bad export cookie 526462594897198927 [ 2607.074773] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2607.075657] LustreError: 156090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2608.044382] LustreError: 154467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.051024] LustreError: 154467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2608.607876] LustreError: 154466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.639883] LustreError: 154466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.237051] Lustre: server umount lustre-MDT0001 complete [ 2615.414105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2619.581800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2619.662865] LustreError: 157999:0:(llog.c:1643:llog_backup()) MGC192.168.203.109@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2619.666806] Lustre: 157999:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.109@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2637.279895] LustreError: 158007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.285244] LustreError: 158007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2642.399903] LustreError: 158007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.404317] LustreError: 158007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2651.616113] LustreError: 158005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.620356] LustreError: 158005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2668.460258] LustreError: 158006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.465784] LustreError: 158006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 2680.415282] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2680.417689] Lustre: 157999:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.109@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2702.303896] LustreError: 158006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.308142] LustreError: 158006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 2709.471210] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2709.473571] Lustre: 154462:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.109@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2721.254933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2721.257536] Lustre: Skipped 2 previous similar messages [ 2722.632991] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2725.601344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.371136] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2726.687132] LustreError: 158726:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2726.689748] LustreError: 158726:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000fe4567ee x1853893420085504/t0(0) o250->MGC192.168.203.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1768010823 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2726.696106] LustreError: 158726:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2727.391699] LustreError: 142274:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d97cd91e x1853893420089856/t0(0) o250->MGC192.168.203.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2728.788390] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2730.642175] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2731.140608] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2732.391420] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2732.518196] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2735.031378] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2740.697645] LustreError: 159483:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2740.700070] LustreError: 159483:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2740.727595] Lustre: server umount lustre-MDT0000 complete [ 2741.937365] LustreError: 154452:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768010838 with bad export cookie 526462594897205381 [ 2741.941694] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2742.752349] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.757859] Lustre: Skipped 4 previous similar messages [ 2742.760655] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2742.763431] Lustre: Skipped 8 previous similar messages [ 2748.075867] Lustre: server umount lustre-MDT0001 complete [ 2749.436809] Lustre: server umount lustre-OST0000 complete [ 2756.395769] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 2759.644068] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2763.566773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2765.843958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2768.192903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2770.596215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2772.968970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2773.000225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.095511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2773.109195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2773.111582] Lustre: Skipped 1 previous similar message [ 2773.151529] Lustre: lustre-MDT0000: new disk, initializing [ 2773.153191] Lustre: Skipped 1 previous similar message [ 2773.183126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2773.186862] Lustre: Skipped 1 previous similar message [ 2773.193545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2773.197146] Lustre: Skipped 1 previous similar message [ 2774.546185] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2778.828499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2778.848557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2778.886387] Lustre: 162826:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2778.989101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2780.292568] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2783.406314] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2784.589440] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2786.856585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2786.880991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2788.796213] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2791.416160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2791.419155] Lustre: Skipped 1 previous similar message [ 2791.421286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2791.445824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2791.847416] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2793.173502] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2793.249368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2794.587146] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2794.660063] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2796.514069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2796.514811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2796.516640] LustreError: Skipped 1 previous similar message [ 2796.517314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2796.522328] Lustre: Skipped 1 previous similar message [ 2796.523668] Lustre: Skipped 1 previous similar message [ 2797.294133] LustreError: 165767:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2797.296362] LustreError: 165767:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2797.331772] Lustre: server umount lustre-OST0000 complete [ 2804.699706] Lustre: server umount lustre-MDT0000 complete [ 2805.878379] LustreError: 164574:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768010902 with bad export cookie 526462594897222160 [ 2805.881760] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2805.883828] LustreError: 164574:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2805.998389] Lustre: server umount lustre-MDT0001 complete [ 2808.892792] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:08:25 (1768010905) [ 2810.970568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2811.134782] LustreError: 166839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2811.139961] LustreError: 166839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 2812.344887] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2815.002713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2816.273901] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2818.011268] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2819.124389] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2821.345331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2823.303247] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2825.013167] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2828.829138] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:08:44 (1768010924) [ 2829.792562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.793701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2829.796306] Lustre: Skipped 3 previous similar messages [ 2829.798564] LustreError: Skipped 1 previous similar message [ 2844.127127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2844.176805] Lustre: server umount lustre-OST0000 complete [ 2846.175893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2851.470462] Lustre: server umount lustre-MDT0000 complete [ 2852.627539] LustreError: 167558:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768010949 with bad export cookie 526462594897223049 [ 2852.630657] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2852.631857] LustreError: 167558:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2852.758621] Lustre: server umount lustre-MDT0001 complete [ 2856.069830] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2857.038059] Key type lgssc unregistered [ 2857.162396] LNet: 170213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2857.164636] LNetError: 170213:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2857.177273] LNet: Removed LNI 192.168.203.109@tcp [ 2857.480945] Key type .llcrypt unregistered [ 2857.482412] Key type ._llcrypt unregistered [ 2864.473232] Key type ._llcrypt registered [ 2864.474410] Key type .llcrypt registered [ 2864.511854] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2864.982164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2864.993421] alg: No test for adler32 (adler32-zlib) [ 2865.848017] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 2865.933682] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2867.519167] Key type lgssc registered [ 2867.847782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2870.282375] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2870.290017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2885.727309] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2890.847191] LustreError: 171413:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.109@tcp: failed processing log, type 4: rc = -110 [ 2895.967365] LustreError: 171413:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2895.973975] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2895.977323] LustreError: MGC192.168.203.109@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2895.982477] LustreError: 171413:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2895.985829] LustreError: 171413:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 2895.988577] LustreError: 171413:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2895.990524] LustreError: 171413:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2896.013338] Lustre: server umount lustre-OST0000 complete [ 2896.015058] LustreError: 171413:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2898.804825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2899.956839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2901.231454] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2903.982558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2904.106210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2905.357836] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2907.067811] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2908.258233] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2910.506051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2910.583269] LustreError: 173504:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2910.586482] Lustre: 173504:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.203.109@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2910.626050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2912.467797] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2914.226835] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2918.882527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.884623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2918.887826] Lustre: Skipped 2 previous similar messages [ 2918.889591] Lustre: Skipped 1 previous similar message [ 2921.018105] LustreError: 174156:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2921.019613] LustreError: 174156:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2921.045531] Lustre: server umount lustre-MDT0000 complete [ 2922.171481] LustreError: 171895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768011018 with bad export cookie 11669191620962154981 [ 2922.174325] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2922.175523] LustreError: 171895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2922.227697] LustreError: 174357:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2922.229753] LustreError: 174357:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2922.268313] Lustre: server umount lustre-MDT0001 complete [ 2932.590504] LustreError: 174558:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2932.592428] LustreError: 174558:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2932.633355] Lustre: server umount lustre-OST0000 complete [ 2935.559417] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 2938.702065] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2941.768496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2943.802920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2945.761147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2947.806164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2950.081968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2950.103184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.189148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2950.202196] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2950.242408] Lustre: lustre-MDT0000: new disk, initializing [ 2950.268463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2950.274630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2951.501520] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2955.360251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2955.380529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2955.412370] Lustre: 176890:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2955.431953] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2955.433445] Lustre: Skipped 1 previous similar message [ 2955.467249] Lustre: lustre-MDT0001: new disk, initializing [ 2955.496149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2955.508618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2955.512425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2956.678044] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2959.607446] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2960.745283] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2962.808785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2962.836523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2962.926447] Lustre: lustre-OST0000: new disk, initializing [ 2962.928600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2964.781972] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2966.199451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2966.202745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2966.231871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2967.815783] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2969.248463] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2969.314575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2970.543716] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2970.609840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2971.617780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.617899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2971.618547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2971.618552] Lustre: Skipped 1 previous similar message [ 2971.621092] Lustre: Skipped 1 previous similar message [ 2971.627234] LustreError: Skipped 1 previous similar message [ 2973.230078] LustreError: 179827:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2973.232404] LustreError: 179827:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2973.268931] Lustre: server umount lustre-OST0000 complete [ 2976.223670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2976.226401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2976.232630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2976.235304] Lustre: Skipped 1 previous similar message [ 2980.625852] Lustre: server umount lustre-MDT0000 complete [ 2981.743152] LustreError: 176881:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768011078 with bad export cookie 11669191620962155681 [ 2981.745531] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2981.746840] LustreError: 176881:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2981.807894] LustreError: 180231:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2981.809471] LustreError: 180231:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2981.852586] Lustre: server umount lustre-MDT0001 complete [ 2984.473336] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 21:11:20 (1768011080) [ 2989.653489] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2990.594965] Key type lgssc unregistered [ 2990.724568] LNet: 181412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2990.727046] LNetError: 181412:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2990.733227] LNet: Removed LNI 192.168.203.109@tcp [ 2990.997202] Key type .llcrypt unregistered [ 2990.998209] Key type ._llcrypt unregistered [ 2997.739927] Key type ._llcrypt registered [ 2997.741141] Key type .llcrypt registered [ 2997.775139] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2998.070278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2998.099810] alg: No test for adler32 (adler32-zlib) [ 2998.938798] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 2999.011308] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 3000.583143] Key type lgssc registered [ 3000.924640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3003.301898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3003.308474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3004.462043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3005.683872] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3008.346851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3008.465820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3009.680213] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3011.362597] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3012.464996] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3014.847139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3014.928972] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3014.930213] LustreError: 184225:0:(llog.c:1684:llog_backup()) MGC192.168.203.109@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3014.933211] Lustre: 184225:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.203.109@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3014.973535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3016.697719] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3018.397706] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3023.330295] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3023.331166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3023.334455] Lustre: Skipped 2 previous similar messages [ 3023.336142] Lustre: Skipped 1 previous similar message [ 3025.145395] LustreError: 184878:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3025.167593] Lustre: server umount lustre-MDT0000 complete [ 3026.274929] LustreError: 182567:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768011122 with bad export cookie 7961014363683826804 [ 3026.277462] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3026.280029] LustreError: 182567:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3026.342980] LustreError: 185081:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3026.345738] LustreError: 185081:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3026.390216] Lustre: server umount lustre-MDT0001 complete [ 3037.809484] LustreError: 185281:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3037.812393] LustreError: 185281:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3037.853357] Lustre: server umount lustre-OST0000 complete [ 3040.672239] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 3043.644864] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3046.639875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3048.704441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3050.689506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3052.652486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3054.818610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3054.839040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.914499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3054.926640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3054.960833] Lustre: lustre-MDT0000: new disk, initializing [ 3054.981849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3054.988931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3056.149219] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3060.017095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3060.037360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3060.066418] Lustre: 187615:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3060.083669] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3060.085604] Lustre: Skipped 1 previous similar message [ 3060.086873] LustreError: 188381:0:(llog.c:1684:llog_backup()) MGC192.168.203.109@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3060.089986] LustreError: 188381:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 3060.092600] Lustre: 188381:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.109@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3060.100433] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3060.102825] Lustre: Skipped 1 previous similar message [ 3060.139285] Lustre: lustre-MDT0001: new disk, initializing [ 3060.167399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3060.178501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3060.181421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3061.365530] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3064.314985] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3065.411785] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3067.429175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3067.450086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3067.528721] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3067.530250] Lustre: Skipped 1 previous similar message [ 3067.531820] LustreError: 189335:0:(llog.c:1684:llog_backup()) MGC192.168.203.109@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3067.535221] LustreError: 189335:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 3067.537706] Lustre: 189335:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.109@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3067.541542] Lustre: 189335:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3067.549082] LustreError: 189345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.553289] Lustre: lustre-OST0000: new disk, initializing [ 3067.555183] LustreError: 189345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3067.556952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3069.302629] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3072.205456] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3072.999722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3073.003127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3073.018284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3073.440459] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3073.500494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3074.687216] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3074.751839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3078.114500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3078.114535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.116504] LustreError: Skipped 1 previous similar message [ 3078.120138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3078.122467] Lustre: Skipped 2 previous similar messages [ 3081.389731] LustreError: 190558:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3081.391500] LustreError: 190558:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3081.426574] Lustre: server umount lustre-OST0000 complete [ 3083.232707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3083.233463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3083.236705] Lustre: Skipped 2 previous similar messages [ 3088.352229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3088.355018] Lustre: Skipped 2 previous similar messages [ 3093.472255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3093.475453] Lustre: Skipped 1 previous similar message [ 3097.055100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3097.089716] LustreError: 190759:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 3097.091946] LustreError: 190759:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3097.118290] Lustre: server umount lustre-MDT0000 complete [ 3097.568262] LustreError: 187625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3097.574189] LustreError: 187625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3098.381415] LustreError: 187606:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768011194 with bad export cookie 7961014363683827504 [ 3098.385087] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3098.386361] LustreError: 187606:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3098.517987] Lustre: server umount lustre-MDT0001 complete [ 3101.458262] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 21:13:17 (1768011197) [ 3106.973543] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3110.514385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.664064] LustreError: 192326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.679836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.682304] Lustre: Skipped 1 previous similar message [ 3112.023418] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3114.827356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3116.219823] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3117.165288] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3119.389167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3121.458300] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3124.349105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3124.374229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3124.434767] Lustre: lustre-OST0001: new disk, initializing [ 3124.437712] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3126.372339] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3130.374384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3130.379060] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3130.408533] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3131.894368] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3134.570290] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3140.187771] Lustre: DEBUG MARKER: ADD OST3 [ 3141.919323] loop: module loaded [ 3141.937650] loop0: detected capacity change from 0 to 204800000 [ 3142.011305] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3142.016572] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3142.021562] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3142.075607] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3144.162992] loop0: detected capacity change from 0 to 204800000 [ 3146.144855] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3146.175466] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3146.249539] Lustre: lustre-OST0003: new disk, initializing [ 3146.252090] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3146.296444] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3146.298936] Lustre: Skipped 3 previous similar messages [ 3147.020176] Lustre: DEBUG MARKER: STOP OST3 [ 3147.661233] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3147.694762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3147.699061] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3147.711538] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3147.731262] Lustre: Failing over lustre-OST0003 [ 3147.763726] LustreError: 196657:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3147.767085] LustreError: 196657:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3147.787041] Lustre: server umount lustre-OST0003 complete [ 3148.154602] LustreError: 193778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.418545] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3149.727141] LustreError: 196626:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3152.153250] Lustre: DEBUG MARKER: START OST3 again [ 3152.865996] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3152.869049] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.872873] LustreError: 193964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3152.872915] Lustre: Skipped 1 previous similar message [ 3154.466086] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3154.572112] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3155.818230] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3155.825759] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3155.825846] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3156.507856] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3170.184370] Lustre: Failing over lustre-OST0003 [ 3170.208145] LustreError: 198617:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3170.210995] LustreError: 198617:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3170.230233] Lustre: server umount lustre-OST0003 complete [ 3170.277264] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3170.283766] LustreError: 193780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3170.284142] Lustre: Skipped 1 previous similar message [ 3170.289527] LustreError: 193780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3185.634662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3185.637231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.638706] Lustre: Skipped 3 previous similar messages [ 3185.641318] Lustre: Skipped 2 previous similar messages [ 3188.971782] Lustre: server umount lustre-MDT0000 complete [ 3190.266688] LustreError: 193428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768011286 with bad export cookie 7961014363683828372 [ 3190.272228] LustreError: 193428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3190.272503] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3190.752483] LustreError: 192322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.757204] LustreError: 192322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3196.535598] Lustre: server umount lustre-MDT0001 complete [ 3197.939826] Lustre: server umount lustre-OST0000 complete [ 3199.314446] Lustre: server umount lustre-OST0001 complete [ 3201.359763] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 3203.902431] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3207.287523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3209.283835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3211.249781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3213.243411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3215.365452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3215.386867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3215.461102] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3215.472132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3215.513914] Lustre: lustre-MDT0000: new disk, initializing [ 3215.535776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3215.537349] Lustre: Skipped 1 previous similar message [ 3215.542477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3216.744708] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3220.719733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3220.743889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3220.773288] Lustre: 201671:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3220.869696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3222.069313] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3224.984949] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3226.078218] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3228.164876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3228.188077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3230.054664] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3230.073319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3230.112106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3233.032475] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3234.286167] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3234.356313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3235.571821] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3235.634123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3238.253390] LustreError: 204606:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3238.255107] LustreError: 204606:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 3238.287546] Lustre: server umount lustre-OST0000 complete [ 3240.416787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.417378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3240.420066] Lustre: Skipped 3 previous similar messages [ 3240.424170] Lustre: Skipped 6 previous similar messages [ 3248.633989] Lustre: server umount lustre-MDT0000 complete [ 3249.846131] LustreError: 201663:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768011346 with bad export cookie 7961014363683830647 [ 3249.849331] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3249.851230] LustreError: 201663:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3260.437298] Lustre: server umount lustre-MDT0001 complete [ 3261.644913] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:15:57 (1768011357) [ 3266.883412] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 3270.309677] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3273.601565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3275.810401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3277.927911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3280.127326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3282.374603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3282.398730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.483226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3282.494538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3282.496309] Lustre: Skipped 3 previous similar messages [ 3282.533795] Lustre: lustre-MDT0000: new disk, initializing [ 3282.535117] Lustre: Skipped 2 previous similar messages [ 3282.558979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3282.561114] Lustre: Skipped 2 previous similar messages [ 3282.567042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3282.571985] Lustre: Skipped 2 previous similar messages [ 3283.949256] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3288.126348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3288.141390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3288.177526] Lustre: 207848:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3288.280926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3289.599559] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3292.739533] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3293.965952] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3296.243626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3296.269522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3297.644193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3298.375611] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3301.359407] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3302.692068] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3302.762180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3304.082131] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3304.152041] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3308.002993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.003220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3308.004546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3308.004549] Lustre: Skipped 2 previous similar messages [ 3308.006924] Lustre: Skipped 1 previous similar message [ 3308.008682] LustreError: Skipped 1 previous similar message [ 3319.263262] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3319.281103] LustreError: 210788:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3319.285306] LustreError: 210788:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3319.325960] Lustre: server umount lustre-OST0000 complete [ 3326.735740] Lustre: server umount lustre-MDT0000 complete [ 3327.931265] LustreError: 207838:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768011424 with bad export cookie 7961014363683831529 [ 3327.934470] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3327.935297] LustreError: 207838:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3328.045511] Lustre: server umount lustre-MDT0001 complete [ 3330.633321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3330.778647] LustreError: 211666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3330.785140] LustreError: 211666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3332.019035] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3334.693619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3336.014357] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3337.616924] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3338.683469] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3340.708769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3342.532071] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3344.128733] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3352.031729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3356.745915] Lustre: server umount lustre-OST0000 complete [ 3364.052862] Lustre: server umount lustre-MDT0000 complete [ 3365.170323] LustreError: 211646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768011461 with bad export cookie 7961014363683832411 [ 3365.173096] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3365.175372] LustreError: 211646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3365.277609] Lustre: server umount lustre-MDT0001 complete [ 3477.495027] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 3478.547054] Key type lgssc unregistered [ 3478.683375] LNet: 214980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3478.685879] LNetError: 214980:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3478.697354] LNet: Removed LNI 192.168.203.109@tcp [ 3479.005147] Key type .llcrypt unregistered [ 3479.006517] Key type ._llcrypt unregistered [ 3486.983614] Key type ._llcrypt registered [ 3486.985148] Key type .llcrypt registered [ 3487.022603] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 3493.207808] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3493.509553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3493.515598] alg: No test for adler32 (adler32-zlib) [ 3494.362721] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 3494.442464] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 3496.023108] Key type lgssc registered [ 3496.366636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3499.314053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3501.453468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3503.433605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3505.576980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3507.778530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3507.798965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3507.803573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3508.888611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3508.900258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3508.939683] Lustre: lustre-MDT0000: new disk, initializing [ 3508.963051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.969121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3510.243858] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3514.351780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3514.377178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3514.413369] Lustre: 218278:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3514.441310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3514.443555] Lustre: Skipped 1 previous similar message [ 3514.483731] Lustre: lustre-MDT0001: new disk, initializing [ 3514.517651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3514.530894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3514.534609] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3515.869895] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3518.915260] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3520.091095] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3522.359726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3522.385569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3522.476808] Lustre: lustre-OST0000: new disk, initializing [ 3522.478931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3522.505729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3524.446099] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3526.268571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3526.272184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3526.309924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3527.554448] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3528.933992] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3529.007952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3530.371507] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3530.446571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3531.745852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3531.745984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3531.749037] LustreError: Skipped 1 previous similar message [ 3531.749702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3531.755964] Lustre: Skipped 1 previous similar message [ 3533.102083] LustreError: 221216:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3533.139709] Lustre: server umount lustre-OST0000 complete [ 3535.327931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3535.330624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.334867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3535.336747] Lustre: Skipped 1 previous similar message [ 3536.864318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3536.864938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3540.544527] LustreError: 221418:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3540.546880] LustreError: 221418:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3540.577623] Lustre: server umount lustre-MDT0000 complete [ 3541.854315] LustreError: 218269:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768011638 with bad export cookie 2972169954072230663 [ 3541.858357] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3541.858533] LustreError: 218269:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3541.924532] LustreError: 221620:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3541.926757] LustreError: 221620:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3541.983941] Lustre: server umount lustre-MDT0001 complete [ 3547.912847] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3550.958070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3551.098985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3552.334016] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3554.984036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3556.220959] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3557.085920] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3559.219182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3559.330155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3559.331890] Lustre: Skipped 1 previous similar message [ 3561.216440] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3564.064625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3564.087424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3564.145635] Lustre: lustre-OST0001: new disk, initializing [ 3564.148181] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3566.074461] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3566.528205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3566.532153] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3566.562949] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3571.460086] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3577.147868] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3585.741391] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:21:21 (1768011681) [ 3587.042943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.044530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3587.051112] Lustre: Skipped 3 previous similar messages [ 3587.056090] Lustre: Skipped 4 previous similar messages [ 3592.161768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3592.163603] Lustre: Skipped 3 previous similar messages [ 3592.511447] LustreError: 226168:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3592.513588] LustreError: 226168:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3592.541987] Lustre: server umount lustre-MDT0000 complete [ 3593.727274] LustreError: 223893:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768011690 with bad export cookie 2972169954072231531 [ 3593.729838] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3593.731915] LustreError: 223893:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3593.841473] Lustre: server umount lustre-MDT0001 complete [ 3605.490565] LustreError: 226570:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3605.492451] LustreError: 226570:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3605.507748] Lustre: server umount lustre-OST0000 complete [ 3616.944683] LustreError: 226772:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3616.946356] LustreError: 226772:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3616.992350] Lustre: server umount lustre-OST0001 complete [ 3618.947207] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 3622.019676] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3625.217967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3627.293418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3629.288753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3631.493361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3633.719725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3633.738899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.823620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3633.835531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3633.875227] Lustre: lustre-MDT0000: new disk, initializing [ 3633.899456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.902668] Lustre: Skipped 1 previous similar message [ 3633.908964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3635.282224] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3639.266833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3639.289433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3639.319012] Lustre: 228953:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3639.420152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3640.726508] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3643.763647] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3644.940815] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3647.140428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3647.164297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3647.267107] Lustre: lustre-OST0000: new disk, initializing [ 3647.268518] Lustre: Skipped 1 previous similar message [ 3647.270257] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3647.271786] Lustre: Skipped 2 previous similar messages [ 3649.088180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3649.090545] Lustre: Skipped 1 previous similar message [ 3649.092615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3649.131738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3649.180361] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3652.276728] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3653.620569] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3653.692689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3654.998639] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3655.070626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3657.710072] LustreError: 231893:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3657.712057] LustreError: 231893:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3657.748733] Lustre: server umount lustre-OST0000 complete [ 3659.232786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.233544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3659.238812] Lustre: Skipped 1 previous similar message [ 3669.500315] Lustre: server umount lustre-MDT0000 complete [ 3670.740327] LustreError: 230699:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768011767 with bad export cookie 2972169954072233631 [ 3670.744222] LustreError: 230699:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3670.745117] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3681.302072] Lustre: server umount lustre-MDT0001 complete [ 3684.317982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.490524] LustreError: 232774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3684.498326] LustreError: 232774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3684.519658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3684.521787] Lustre: Skipped 2 previous similar messages [ 3685.876903] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3688.652850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3690.060550] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3691.785495] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3692.998661] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3695.220392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3697.216186] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3698.969883] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3848.161578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3848.162071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3848.164452] LustreError: Skipped 1 previous similar message [ 3848.164991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3848.168154] Lustre: Skipped 1 previous similar message [ 3848.173016] Lustre: Skipped 6 previous similar messages [ 3849.964185] LustreError: 235060:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3849.966053] LustreError: 235060:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3850.002308] Lustre: server umount lustre-OST0000 complete [ 3852.767943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3857.372535] Lustre: server umount lustre-MDT0000 complete [ 3858.399935] LustreError: 232774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3858.405219] LustreError: 232774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3858.570341] LustreError: 232755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768011955 with bad export cookie 2972169954072234513 [ 3858.573185] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3858.573661] LustreError: 232755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3858.688209] Lustre: server umount lustre-MDT0001 complete [ 3961.698909] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 3962.772451] Key type lgssc unregistered [ 3962.915317] LNet: 236029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3962.917621] LNetError: 236029:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3962.924366] LNet: Removed LNI 192.168.203.109@tcp [ 3963.212089] Key type .llcrypt unregistered [ 3963.213252] Key type ._llcrypt unregistered [ 3971.441869] Key type ._llcrypt registered [ 3971.442942] Key type .llcrypt registered [ 3971.480779] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 3977.502508] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3977.851574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3977.864360] alg: No test for adler32 (adler32-zlib) [ 3978.707960] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 3978.783471] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 3980.359113] Key type lgssc registered [ 3980.695845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3983.575247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3985.685057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3987.697486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3989.787903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3995.325370] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3998.662589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3998.683559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3998.689929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3999.774646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3999.789146] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3999.825145] Lustre: lustre-MDT0000: new disk, initializing [ 3999.847940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3999.854461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4001.164831] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4005.189780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4005.219804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4005.251854] Lustre: 240025:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4005.276968] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4005.279882] Lustre: Skipped 1 previous similar message [ 4005.317080] Lustre: lustre-MDT0001: new disk, initializing [ 4005.348801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4005.359436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4005.362780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4006.599358] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4008.876559] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4011.025930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4011.048882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4011.144015] Lustre: lustre-OST0000: new disk, initializing [ 4011.146948] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4011.178574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4013.020995] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4014.975428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4014.979980] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4015.042708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4017.252631] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4017.287983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4017.355707] Lustre: lustre-OST0001: new disk, initializing [ 4017.357587] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4017.398893] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4019.265370] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4024.573198] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4025.345243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4025.350709] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4025.381869] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4029.158134] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4040.675409] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4040.676297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4040.678981] Lustre: Skipped 3 previous similar messages [ 4040.680957] Lustre: Skipped 2 previous similar messages [ 4043.013273] LustreError: 243551:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4043.042397] Lustre: server umount lustre-MDT0000 complete [ 4044.237310] LustreError: 240805:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768012140 with bad export cookie 9152602373933986216 [ 4044.240445] LustreError: 240805:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4044.240786] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4044.315535] LustreError: 243753:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4044.318199] LustreError: 243753:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4044.364586] Lustre: server umount lustre-MDT0001 complete [ 4055.988115] LustreError: 243953:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4055.991362] LustreError: 243953:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4056.011390] Lustre: server umount lustre-OST0000 complete [ 4067.379822] LustreError: 244155:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4067.382020] LustreError: 244155:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4067.423376] Lustre: server umount lustre-OST0001 complete [ 4071.658632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4074.233967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4074.553313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4077.287192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4080.076861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4080.399048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4086.314017] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4089.623209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4089.629055] Lustre: lustre-MDT0000: reset Object Index mappings [ 4089.796889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4091.010423] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4093.676221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4093.680761] Lustre: lustre-MDT0001: reset Object Index mappings [ 4095.029475] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4095.911699] Lustre: 248572:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4097.984303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4098.101031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4098.102892] Lustre: Skipped 1 previous similar message [ 4099.852415] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4100.132317] LustreError: 248927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4100.137375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4100.138677] LustreError: 248927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4102.585230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4103.726777] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4104.621406] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4107.692707] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4108.942154] Lustre: 250419:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4119.011792] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4119.013319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4119.015389] Lustre: Skipped 2 previous similar messages [ 4119.018255] Lustre: Skipped 4 previous similar messages [ 4121.149734] LustreError: 250523:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4121.152651] LustreError: 250523:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4121.179729] Lustre: server umount lustre-MDT0000 complete [ 4122.346399] LustreError: 247441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768012218 with bad export cookie 9152602373933992992 [ 4122.349119] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4122.352504] LustreError: 247441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4122.466643] Lustre: server umount lustre-MDT0001 complete [ 4134.324104] LustreError: 250926:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4134.326222] LustreError: 250926:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4134.342754] Lustre: server umount lustre-OST0000 complete [ 4144.724204] Lustre: server umount lustre-OST0001 complete [ 4150.814361] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:30:46 (1768012246) [ 4156.047788] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 4158.924688] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4162.196575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4164.317565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4166.229383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4168.243109] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4170.512606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4170.534374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4170.614809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4170.625144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4170.662215] Lustre: lustre-MDT0000: new disk, initializing [ 4170.685576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4170.687592] Lustre: Skipped 1 previous similar message [ 4170.692625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4171.975682] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4176.161639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4176.190542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4176.239058] Lustre: 254644:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4176.370882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4177.701342] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4181.026945] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4182.267077] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4184.384215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4184.404254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4184.493446] Lustre: lustre-OST0000: new disk, initializing [ 4184.494718] Lustre: Skipped 1 previous similar message [ 4184.496305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4184.497937] Lustre: Skipped 2 previous similar messages [ 4185.848203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4185.850740] Lustre: Skipped 1 previous similar message [ 4185.852606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4185.885617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4186.335687] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4189.431871] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4190.891104] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4190.979184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4192.321394] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4192.397207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4195.120076] LustreError: 257584:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4195.122125] LustreError: 257584:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4195.158879] Lustre: server umount lustre-OST0000 complete [ 4196.831885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4196.834316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.837552] Lustre: Skipped 1 previous similar message [ 4196.839121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4201.441427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4201.442602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.561500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4206.564313] Lustre: Skipped 2 previous similar messages [ 4206.596924] Lustre: server umount lustre-MDT0000 complete [ 4207.828164] LustreError: 254634:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768012304 with bad export cookie 9152602373933995085 [ 4207.831447] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4207.832946] LustreError: 254634:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4218.390637] Lustre: server umount lustre-MDT0001 complete [ 4224.253870] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4227.683065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.824305] LustreError: 259160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.840067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4227.841788] Lustre: Skipped 2 previous similar messages [ 4229.157502] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4231.982836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4233.387678] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4234.352657] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4236.522650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4238.441824] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4241.203948] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4241.229648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4241.289137] Lustre: lustre-OST0001: new disk, initializing [ 4241.290957] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4242.428296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4242.431552] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4242.467657] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4243.192194] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4247.508993] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4248.645936] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4255.313839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4255.316848] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4257.764061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4257.765749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4257.766068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4257.769786] Lustre: Skipped 3 previous similar messages [ 4263.230184] LustreError: 262662:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4263.232048] LustreError: 262662:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4263.263209] Lustre: server umount lustre-MDT0000 complete [ 4264.481591] LustreError: 260260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768012361 with bad export cookie 9152602373933995967 [ 4264.485333] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4264.488172] LustreError: 260260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4264.613250] Lustre: server umount lustre-MDT0001 complete [ 4276.229221] Lustre: server umount lustre-OST0000 complete [ 4287.774149] Lustre: server umount lustre-OST0001 complete [ 4293.063530] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4296.628258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4296.773532] LustreError: 264440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4296.796096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4296.798833] Lustre: Skipped 3 previous similar messages [ 4298.317054] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4301.467132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4303.105776] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4304.074146] Lustre: 265551:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4306.241547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4307.426857] LustreError: 265906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.434364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4308.212856] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4311.000187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4312.108758] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4312.998657] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4316.164799] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4317.290692] Lustre: 267404:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4323.929789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4323.937219] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4326.881234] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:33:43 (1768012423) [ 4342.755781] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.756109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4342.757638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4342.757642] Lustre: Skipped 7 previous similar messages [ 4342.759174] Lustre: Skipped 3 previous similar messages [ 4344.514322] LustreError: 268113:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4344.516500] LustreError: 268113:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4344.549283] Lustre: server umount lustre-MDT0000 complete [ 4345.776280] LustreError: 264421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768012442 with bad export cookie 9152602373934002260 [ 4345.780201] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4345.781441] LustreError: 264421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4347.871993] LustreError: 264436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4347.876832] LustreError: 264436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4356.001096] LustreError: 264436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.009467] LustreError: 264436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4360.159081] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4360.243498] Lustre: server umount lustre-MDT0001 complete [ 4367.552585] Lustre: server umount lustre-OST0000 complete [ 4383.199080] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4383.261324] Lustre: server umount lustre-OST0001 complete [ 4385.270578] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 4388.816926] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4392.233794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4394.360233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4396.371285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4398.432520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4403.573752] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4406.878033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4406.900568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4406.987448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4406.998865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4407.040662] Lustre: lustre-MDT0000: new disk, initializing [ 4407.075106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4408.426716] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4412.566015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4412.588799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4412.620892] Lustre: 271597:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4412.725861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4413.963239] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4418.389357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4418.415170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4419.999184] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4420.405532] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4424.614395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4424.638380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4426.026272] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4426.542856] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4431.031379] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4441.408587] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 21:35:37 (1768012537) [ 4443.615816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4443.618164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4443.622556] Lustre: Skipped 2 previous similar messages [ 4443.624210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4443.627034] Lustre: Skipped 13 previous similar messages [ 4448.218501] Lustre: server umount lustre-MDT0000 complete [ 4449.463424] LustreError: 274056:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768012545 with bad export cookie 9152602373934010128 [ 4449.468060] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4449.600650] Lustre: server umount lustre-MDT0001 complete [ 4461.064692] Lustre: server umount lustre-OST0000 complete [ 4471.576613] Lustre: server umount lustre-OST0001 complete [ 4477.548793] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 4480.916968] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4484.025617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4486.155907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4488.227514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4490.380518] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4492.899082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4492.919631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4492.996439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4492.999276] Lustre: Skipped 2 previous similar messages [ 4493.007541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4493.009292] Lustre: Skipped 4 previous similar messages [ 4493.046797] Lustre: lustre-MDT0000: new disk, initializing [ 4493.048027] Lustre: Skipped 3 previous similar messages [ 4493.071458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4493.073280] Lustre: Skipped 7 previous similar messages [ 4493.078286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4493.080368] Lustre: Skipped 3 previous similar messages [ 4494.306021] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4498.312944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4498.341090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4498.372433] Lustre: 278650:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4498.493487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4498.495918] Lustre: Skipped 2 previous similar messages [ 4499.776100] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4502.918241] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4504.096822] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4506.422972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4506.447993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4508.384382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4508.453261] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4511.501358] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4512.195179] Lustre: Mounted lustre-client [ 4512.528762] LustreError: 281154:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8b2c45ccf800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4512.534876] LustreError: 281154:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4512.537259] LustreError: 281154:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 4512.565848] Lustre: Unmounted lustre-client [ 4517.346333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4517.347348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4517.349586] Lustre: Skipped 4 previous similar messages [ 4517.351455] Lustre: Skipped 4 previous similar messages [ 4521.553238] Lustre: server umount lustre-OST0000 complete [ 4523.999840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4537.311072] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4537.369398] Lustre: server umount lustre-MDT0000 complete [ 4538.641039] LustreError: 278642:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768012635 with bad export cookie 9152602373934011654 [ 4538.643867] LustreError: 278642:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4538.644415] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4538.777945] Lustre: server umount lustre-MDT0001 complete [ 4542.246172] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4543.290709] Key type lgssc unregistered [ 4569.631117] LNet: 1 local NIs in recovery (showing 1): 192.168.203.109@tcp [ 4570.079323] LNet: 282432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4570.083206] LNetError: 282432:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4642.789439] LNet: Removed LNI 192.168.203.109@tcp [ 4643.126138] Key type .llcrypt unregistered [ 4643.128102] Key type ._llcrypt unregistered [ 4648.911707] Key type ._llcrypt registered [ 4648.913114] Key type .llcrypt registered [ 4648.951551] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 4655.992841] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4656.297904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4656.306469] alg: No test for adler32 (adler32-zlib) [ 4657.175150] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 4657.261801] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 4658.839145] Key type lgssc registered [ 4659.158921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4661.993747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4664.147967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4666.131774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4668.449170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4669.080782] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:39:25 (1768012765) [ 4671.234844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4671.255298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4671.260237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4672.351058] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4672.364812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4672.403038] Lustre: lustre-MDT0000: new disk, initializing [ 4672.426262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4672.432270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4673.676380] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4677.772204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4677.795405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4677.826233] Lustre: 285601:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4677.851191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4677.852768] Lustre: Skipped 1 previous similar message [ 4677.891248] Lustre: lustre-MDT0001: new disk, initializing [ 4677.926382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4677.943385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4677.946929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4679.231303] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4682.639843] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4683.903578] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4686.261333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4686.293215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4686.413607] Lustre: lustre-OST0000: new disk, initializing [ 4686.415753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4686.456633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4688.336855] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4688.885817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4688.888679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4688.928924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4691.405147] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4697.260611] LustreError: 288043:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4697.294706] Lustre: server umount lustre-OST0000 complete [ 4698.591981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4698.594219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4698.598230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4699.105270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4699.106186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4704.224234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4704.226317] Lustre: Skipped 2 previous similar messages [ 4709.345772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4709.348231] Lustre: Skipped 1 previous similar message [ 4709.350541] LustreError: 288243:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4709.354210] LustreError: 288243:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4709.385593] Lustre: server umount lustre-MDT0000 complete [ 4710.632204] LustreError: 286381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768012807 with bad export cookie 10425661307510374537 [ 4710.635336] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4710.635432] LustreError: 286381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4721.124389] LustreError: 288446:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4721.127704] LustreError: 288446:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4721.177990] Lustre: server umount lustre-MDT0001 complete [ 4724.521734] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4725.427577] Key type lgssc unregistered [ 4725.545354] LNet: 288978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4725.547567] LNetError: 288978:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4725.557324] LNet: Removed LNI 192.168.203.109@tcp [ 4725.850376] Key type .llcrypt unregistered [ 4725.851597] Key type ._llcrypt unregistered [ 4759.101736] Key type ._llcrypt registered [ 4759.103049] Key type .llcrypt registered [ 4759.140746] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4759.243373] Key type .llcrypt unregistered [ 4759.244430] Key type ._llcrypt unregistered [ 4786.067673] Key type ._llcrypt registered [ 4786.068814] Key type .llcrypt registered [ 4786.107363] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4786.208204] Key type .llcrypt unregistered [ 4786.209415] Key type ._llcrypt unregistered [ 4796.891769] Key type ._llcrypt registered [ 4796.892868] Key type .llcrypt registered [ 4796.932512] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4797.021641] Key type .llcrypt unregistered [ 4797.022813] Key type ._llcrypt unregistered [ 4807.916891] Key type ._llcrypt registered [ 4807.918767] Key type .llcrypt registered [ 4807.959641] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4808.062524] Key type .llcrypt unregistered [ 4808.064325] Key type ._llcrypt unregistered [ 4835.719293] Key type ._llcrypt registered [ 4835.720983] Key type .llcrypt registered [ 4835.762067] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4835.858442] Key type .llcrypt unregistered [ 4835.860304] Key type ._llcrypt unregistered [ 4846.924376] Key type ._llcrypt registered [ 4846.925424] Key type .llcrypt registered [ 4846.964784] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4847.074187] Key type .llcrypt unregistered [ 4847.075305] Key type ._llcrypt unregistered [ 4858.799371] Key type ._llcrypt registered [ 4858.800818] Key type .llcrypt registered [ 4858.851856] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4858.987528] Key type .llcrypt unregistered [ 4858.989883] Key type ._llcrypt unregistered [ 4870.527687] Key type ._llcrypt registered [ 4870.528881] Key type .llcrypt registered [ 4870.567189] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4870.664261] Key type .llcrypt unregistered [ 4870.665606] Key type ._llcrypt unregistered [ 4882.109241] Key type ._llcrypt registered [ 4882.110298] Key type .llcrypt registered [ 4882.150450] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4882.273240] Key type .llcrypt unregistered [ 4882.274920] Key type ._llcrypt unregistered [ 4892.731386] Key type ._llcrypt registered [ 4892.732457] Key type .llcrypt registered [ 4892.766243] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4892.853159] Key type .llcrypt unregistered [ 4892.854195] Key type ._llcrypt unregistered [ 4913.449030] Key type ._llcrypt registered [ 4913.450179] Key type .llcrypt registered [ 4913.486327] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4913.587261] Key type .llcrypt unregistered [ 4913.589320] Key type ._llcrypt unregistered [ 4924.335488] Key type ._llcrypt registered [ 4924.336459] Key type .llcrypt registered [ 4924.376756] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4924.476221] Key type .llcrypt unregistered [ 4924.477726] Key type ._llcrypt unregistered [ 4936.379273] Key type ._llcrypt registered [ 4936.380668] Key type .llcrypt registered [ 4936.422220] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4936.527215] Key type .llcrypt unregistered [ 4936.528321] Key type ._llcrypt unregistered [ 4958.468286] Key type ._llcrypt registered [ 4958.469611] Key type .llcrypt registered [ 4958.511675] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 4964.583725] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4964.939706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4964.956664] alg: No test for adler32 (adler32-zlib) [ 4965.807841] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 4965.890886] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 4967.479130] Key type lgssc registered [ 4967.834534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4970.803098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4973.707505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4975.869918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4978.355745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4978.935388] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4811 sec ========== 21:44:35 (1768013075) [ 4979.483150] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:44:35 (1768013075) === [ 4980.575136] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:44:36 (1768013076) === [ 4994.732907] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4995.714164] Key type lgssc unregistered [ 4995.852551] LNet: 310682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4995.856183] LNetError: 310682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4995.867448] LNet: Removed LNI 192.168.203.109@tcp [ 4996.180362] Key type .llcrypt unregistered [ 4996.181353] Key type ._llcrypt unregistered