[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 660709362 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002437] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004021] kvm-guest: setup PV IPIs [ 0.009345] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011017] pid_max: default: 32768 minimum: 301 [ 0.012178] LSM: Security Framework initializing [ 0.013135] Yama: becoming mindful. [ 0.014057] SELinux: Initializing. [ 0.015108] *** VALIDATE selinux *** [ 0.023951] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029330] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030213] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031131] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033132] *** VALIDATE tmpfs *** [ 0.035186] *** VALIDATE proc *** [ 0.037316] *** VALIDATE cgroup *** [ 0.038009] *** VALIDATE cgroup2 *** [ 0.039380] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041252] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042031] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043030] Spectre V2 : User space: Vulnerable [ 0.044010] Speculative Store Bypass: Vulnerable [ 0.048127] debug: unmapping init [mem 0xffffffffb2a59000-0xffffffffb2a60fff] [ 0.051228] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.053029] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.054076] ... version: 2 [ 0.055088] ... bit width: 48 [ 0.056032] ... generic registers: 4 [ 0.057026] ... value mask: 0000ffffffffffff [ 0.058013] ... max period: 00007fffffffffff [ 0.059015] ... fixed-purpose events: 3 [ 0.060052] ... event mask: 000000070000000f [ 0.061351] rcu: Hierarchical SRCU implementation. [ 0.064077] smp: Bringing up secondary CPUs ... [ 0.065643] x86: Booting SMP configuration: [ 0.066044] .... node #0, CPUs: #1 #2 #3 [ 0.073476] smp: Brought up 1 node, 4 CPUs [ 0.075014] smpboot: Max logical packages: 1 [ 0.076051] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.185962] node 0 deferred pages initialised in 102ms [ 0.190010] devtmpfs: initialized [ 0.191323] x86/mm: Memory block size: 128MB [ 0.194945] gcov: version magic: 0x41383552 [ 0.197008] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.198116] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.199549] pinctrl core: initialized pinctrl subsystem [ 0.200287] [ 0.201007] ************************************************************* [ 0.202017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.203011] ** ** [ 0.204023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.205020] ** ** [ 0.206010] ** This means that this kernel is built to expose internal ** [ 0.207017] ** IOMMU data structures, which may compromise security on ** [ 0.208018] ** your system. ** [ 0.209010] ** ** [ 0.210011] ** If you see this message and you are not debugging the ** [ 0.211007] ** kernel, report this immediately to your vendor! ** [ 0.212009] ** ** [ 0.213011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.214011] ************************************************************* [ 0.215901] NET: Registered protocol family 16 [ 0.216412] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.217041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.218038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.220125] cpuidle: using governor menu [ 0.227235] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.230866] PCI: Using configuration type 1 for base access [ 0.234239] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.245068] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.246018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.248119] cryptd: max_cpu_qlen set to 1000 [ 0.250429] ACPI: Added _OSI(Module Device) [ 0.251013] ACPI: Added _OSI(Processor Device) [ 0.252011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.253008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.257011] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.261000] ACPI: Interpreter enabled [ 0.262042] ACPI: PM: (supports S0 S3 S4 S5) [ 0.264009] ACPI: Using IOAPIC for interrupt routing [ 0.265102] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.266410] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.274000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.276025] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.279015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.285000] acpiphp: Slot [2] registered [ 0.285000] acpiphp: Slot [3] registered [ 0.286058] acpiphp: Slot [4] registered [ 0.288071] acpiphp: Slot [5] registered [ 0.289104] acpiphp: Slot [6] registered [ 0.290098] acpiphp: Slot [7] registered [ 0.292075] acpiphp: Slot [8] registered [ 0.293008] acpiphp: Slot [9] registered [ 0.293887] acpiphp: Slot [10] registered [ 0.294090] acpiphp: Slot [11] registered [ 0.295063] acpiphp: Slot [12] registered [ 0.297068] acpiphp: Slot [13] registered [ 0.298064] acpiphp: Slot [14] registered [ 0.299056] acpiphp: Slot [15] registered [ 0.300056] acpiphp: Slot [16] registered [ 0.301060] acpiphp: Slot [17] registered [ 0.302098] acpiphp: Slot [18] registered [ 0.303073] acpiphp: Slot [19] registered [ 0.305064] acpiphp: Slot [20] registered [ 0.306072] acpiphp: Slot [21] registered [ 0.307000] acpiphp: Slot [22] registered [ 0.307000] acpiphp: Slot [23] registered [ 0.307089] acpiphp: Slot [24] registered [ 0.308000] acpiphp: Slot [25] registered [ 0.309044] acpiphp: Slot [26] registered [ 0.310064] acpiphp: Slot [27] registered [ 0.311065] acpiphp: Slot [28] registered [ 0.312069] acpiphp: Slot [29] registered [ 0.313083] acpiphp: Slot [30] registered [ 0.315110] acpiphp: Slot [31] registered [ 0.316056] PCI host bridge to bus 0000:00 [ 0.318016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.319012] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.321012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.324014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.326011] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.328014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329204] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.333701] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.337550] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.346748] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.352054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.355022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.357017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.359014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.361455] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.364623] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.366022] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.369627] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.372857] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.383000] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.387011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.391317] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417032] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.439014] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.450023] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.463017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.483000] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.505427] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.513029] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.519026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.533017] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.548000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.560029] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.570014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.590044] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.608000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.610029] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.618014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.628018] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.637909] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.646021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.652011] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.673011] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.687580] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.694157] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.697745] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.701778] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.705338] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.714181] iommu: Default domain type: Passthrough [ 0.715627] SCSI subsystem initialized [ 0.716807] ACPI: bus type USB registered [ 0.720263] usbcore: registered new interface driver usbfs [ 0.721000] usbcore: registered new interface driver hub [ 0.722142] usbcore: registered new device driver usb [ 0.725333] pps_core: LinuxPPS API ver. 1 registered [ 0.727009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.732050] PTP clock support registered [ 0.736036] EDAC MC: Ver: 3.0.0 [ 0.740150] PCI: Using ACPI for IRQ routing [ 0.746305] NetLabel: Initializing [ 0.747013] NetLabel: domain hash size = 128 [ 0.750008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.756121] NetLabel: unlabeled traffic allowed by default [ 0.757043] vgaarb: loaded [ 0.758310] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.759013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.765942] clocksource: Switched to clocksource kvm-clock [ 0.953942] VFS: Disk quotas dquot_6.6.0 [ 0.955641] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.958411] *** VALIDATE ramfs *** [ 0.959647] *** VALIDATE hugetlbfs *** [ 0.961734] pnp: PnP ACPI init [ 0.964321] pnp: PnP ACPI: found 6 devices [ 0.983858] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.987495] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.989898] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.992475] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.995096] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.998332] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.001648] NET: Registered protocol family 2 [ 1.004160] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.010258] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.013502] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.018860] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.024174] TCP: Hash tables configured (established 65536 bind 65536) [ 1.027941] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.031153] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.034556] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.038089] NET: Registered protocol family 1 [ 1.044028] RPC: Registered named UNIX socket transport module. [ 1.049255] RPC: Registered udp transport module. [ 1.052373] RPC: Registered tcp transport module. [ 1.055256] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.058682] NET: Registered protocol family 44 [ 1.061487] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.067446] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.069825] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.072866] PCI: CLS 0 bytes, default 64 [ 1.074538] Unpacking initramfs... [ 3.162662] debug: unmapping init [mem 0xffff90653cc54000-0xffff90653ffbffff] [ 3.168162] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.170133] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.173481] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.841593] Initialise system trusted keyrings [ 3.843400] Key type blacklist registered [ 3.845625] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.854866] zbud: loaded [ 3.858616] *** VALIDATE nfs *** [ 3.859910] *** VALIDATE nfs4 *** [ 3.861524] pstore: using deflate compression [ 3.865264] Platform Keyring initialized [ 4.053967] NET: Registered protocol family 38 [ 4.055798] Key type asymmetric registered [ 4.057752] Asymmetric key parser 'x509' registered [ 4.063983] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.067508] io scheduler mq-deadline registered [ 4.071814] io scheduler kyber registered [ 4.073731] io scheduler bfq registered [ 4.075630] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.082264] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.085443] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.088335] ACPI: Power Button [PWRF] [ 4.244256] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.403947] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.658946] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.769833] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.994942] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.028667] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.067859] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.072378] Non-volatile memory driver v1.3 [ 5.073731] Linux agpgart interface v0.103 [ 5.109699] virtio_blk virtio1: [vda] 134040 512-byte logical blocks (68.6 MB/65.4 MiB) [ 5.112089] vda: detected capacity change from 0 to 68628480 [ 5.129473] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.132618] vdb: detected capacity change from 0 to 1073741824 [ 5.153192] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.156776] vdc: detected capacity change from 0 to 2621440000 [ 5.174519] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.177428] vdd: detected capacity change from 0 to 2621440000 [ 5.194811] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.198943] vde: detected capacity change from 0 to 4294967296 [ 5.222637] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.226914] vdf: detected capacity change from 0 to 4294967296 [ 5.235925] libphy: Fixed MDIO Bus: probed [ 5.247154] usbcore: registered new interface driver usbserial_generic [ 5.249520] usbserial: USB Serial support registered for generic [ 5.255561] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.260318] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.261983] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.264374] mousedev: PS/2 mouse device common for all mice [ 5.266976] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.270376] rtc_cmos 00:05: RTC can wake from S4 [ 5.274198] rtc_cmos 00:05: registered as rtc0 [ 5.275510] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.278319] intel_pstate: CPU model not supported [ 5.281851] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.284963] hid: raw HID events driver (C) Jiri Kosina [ 5.286238] usbcore: registered new interface driver usbhid [ 5.287245] usbhid: USB HID core driver [ 5.288134] drop_monitor: Initializing network drop monitor service [ 5.289775] Initializing XFRM netlink socket [ 5.291233] NET: Registered protocol family 10 [ 5.294163] Segment Routing with IPv6 [ 5.295461] NET: Registered protocol family 17 [ 5.299670] mpls_gso: MPLS GSO support [ 5.304808] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.307719] hpet1: lost 1 rtc interrupts [ 5.308507] RAS: Correctable Errors collector initialized. [ 5.308534] AVX version of gcm_enc/dec engaged. [ 5.308536] AES CTR mode by8 optimization enabled [ 5.430650] sched_clock: Marking stable (5430633664, 0)->(6614604093, -1183970429) [ 5.438574] registered taskstats version 1 [ 5.440763] Loading compiled-in X.509 certificates [ 5.442993] zswap: loaded using pool lzo/zbud [ 5.494589] Key type big_key registered [ 5.522159] Key type encrypted registered [ 5.526012] ima: No TPM chip found, activating TPM-bypass! [ 5.527885] ima: Allocated hash algorithm: sha1 [ 5.530355] ima: No architecture policies found [ 5.532507] evm: Initialising EVM extended attributes: [ 5.537823] evm: security.selinux [ 5.538782] evm: security.ima [ 5.539964] evm: security.capability [ 5.540939] evm: HMAC attrs: 0x1 [ 5.544721] rtc_cmos 00:05: setting system clock to 2025-12-29 18:54:58 UTC (1767034498) [ 5.556208] debug: unmapping init [mem 0xffffffffb3a03000-0xffffffffb3bfffff] [ 5.559318] debug: unmapping init [mem 0xffffffffb2782000-0xffffffffb2a58fff] [ 5.566157] Write protecting the kernel read-only data: 28672k [ 5.572552] debug: unmapping init [mem 0xffffffffb0e03000-0xffffffffb0ffffff] [ 5.574894] debug: unmapping init [mem 0xffffffffb1714000-0xffffffffb17fffff] [ 5.644400] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.657599] systemd[1]: Detected virtualization kvm. [ 5.660287] systemd[1]: Detected architecture x86-64. [ 5.669317] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.697642] systemd[1]: No hostname configured. [ 5.699677] systemd[1]: Set hostname to . [ 5.703678] random: systemd: uninitialized urandom read (16 bytes read) [ 5.705807] systemd[1]: Initializing machine ID from random generator. [ 5.834950] random: ln: uninitialized urandom read (6 bytes read) [ 6.061839] random: systemd: uninitialized urandom read (16 bytes read) [ 6.068714] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.072986] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 6.082482] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.765613] device-mapper: uevent: version 1.0.3 [ 7.770334] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.127732] virtio_net virtio0 ens2: renamed from eth0 [ 9.341242] random: fast init done [ 9.594222] scsi host0: ata_piix [ 9.732829] scsi host1: ata_piix [ 9.738978] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.753989] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.078983] random: crng init done [ 15.080199] random: 7 urandom warning(s) missed due to ratelimiting [ 16.471435] dracut-initqueue[597]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.134172] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.490659] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.989814] SELinux: Disabled at runtime. [ 20.094230] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.109862] systemd[1]: Detected virtualization kvm. [ 20.112668] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.604568] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.611822] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.633477] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.645778] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.659936] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.671518] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.681274] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ 21.897958] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... Starting udev Coldplug all Devices... [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 22.769397] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.102200] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.172752] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.005915] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.522452] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 29.424243] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 30.488221] NFS: Registering the id_resolver key type [ 30.497609] Key type id_resolver registered [ 30.504724] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg413-server login: [ 63.623430] libcfs: loading out-of-tree module taints kernel. [ 63.640255] Key type ._llcrypt registered [ 63.641316] Key type .llcrypt registered [ 63.680342] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 70.919806] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 71.520508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 71.532060] alg: No test for adler32 (adler32-zlib) [ 72.506299] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 72.798398] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 74.432148] Key type lgssc registered [ 74.979694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.614458] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 82.055846] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 85.342863] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 88.327312] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.345456] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 97.867530] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 103.363024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.388808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 103.396203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 104.486943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 104.502348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 104.546654] Lustre: lustre-MDT0000: new disk, initializing [ 104.577903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 104.587457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 106.062658] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.472205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.506581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 111.545913] Lustre: 6495:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 111.566449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 111.569295] Lustre: Skipped 1 previous similar message [ 111.612737] Lustre: lustre-MDT0001: new disk, initializing [ 111.649662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 111.667089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 111.671628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 113.277351] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 116.129806] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 122.587780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.680144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 123.004381] Lustre: lustre-OST0000: new disk, initializing [ 123.014143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 123.094274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 127.500431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 127.511445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 127.615676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 127.986010] hrtimer: interrupt took 4000863 ns [ 128.227529] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 138.210384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 138.294474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 138.385219] Lustre: lustre-OST0001: new disk, initializing [ 138.389641] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 138.445571] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 143.521715] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 145.424589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 145.428944] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 145.501684] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 151.326130] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 156.477444] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 163.751192] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing check_logdir /tmp/testlogs/ [ 165.700571] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing yml_node [ 168.228283] Lustre: DEBUG MARKER: Client: 2.17.0.RC3 [ 169.386881] Lustre: DEBUG MARKER: MDS: 2.17.0.RC3 [ 170.863146] Lustre: DEBUG MARKER: OSS: 2.17.0.RC3 [ 171.707886] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Dec 29 13:57:44 EST 2025 [ 179.896539] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 180.600444] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 193.504929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 193.507692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 193.511915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 196.577851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 196.578151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 196.584205] Lustre: Skipped 1 previous similar message [ 196.587041] Lustre: Skipped 1 previous similar message [ 201.697108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 201.701415] Lustre: Skipped 3 previous similar messages [ 206.817033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 206.819885] Lustre: Skipped 3 previous similar messages [ 207.328641] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 207.381952] LustreError: 12195:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 207.436829] Lustre: server umount lustre-MDT0000 complete [ 209.392902] LustreError: 6487:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767034702 with bad export cookie 4803680301519253463 [ 209.393473] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 209.400207] LustreError: 6487:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 211.939514] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 211.940458] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 211.946570] Lustre: Skipped 2 previous similar messages [ 215.562503] LustreError: 12400:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 215.565753] LustreError: 12400:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 215.687921] Lustre: server umount lustre-MDT0001 complete [ 218.131177] LustreError: 12602:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 218.138061] LustreError: 12602:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 218.170589] Lustre: server umount lustre-OST0000 complete [ 220.396164] LustreError: 12802:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 220.399942] LustreError: 12802:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 220.484323] Lustre: server umount lustre-OST0001 complete [ 223.998638] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 228.711346] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 234.326775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 237.966040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 241.391926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 244.662839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 248.417619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 248.450992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 248.604925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 248.624725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 248.678432] Lustre: lustre-MDT0000: new disk, initializing [ 248.730547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 248.741786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 250.604743] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 256.636418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 256.679994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 256.743805] Lustre: 14982:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 256.786566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 256.789479] Lustre: Skipped 1 previous similar message [ 256.832081] Lustre: lustre-MDT0001: new disk, initializing [ 256.901423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 256.922677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 256.927523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 258.985248] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 263.106565] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 264.754236] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 267.938188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 267.976719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 269.584049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 269.666362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 270.866227] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 274.750076] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 276.631359] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 276.744616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 278.555446] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 278.664910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 280.034639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 280.035377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.038436] LustreError: Skipped 1 previous similar message [ 280.047776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 280.050502] Lustre: Skipped 4 previous similar messages [ 285.682874] LustreError: 17921:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 285.685293] LustreError: 17921:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 285.768123] Lustre: server umount lustre-OST0000 complete [ 290.275470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.283828] Lustre: Skipped 2 previous similar messages [ 293.985612] LustreError: 18122:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 293.995788] LustreError: 18122:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 294.082152] Lustre: server umount lustre-MDT0000 complete [ 295.394191] LustreError: 14990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 295.399222] LustreError: 14990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 296.182333] LustreError: 15762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767034789 with bad export cookie 4803680301519254982 [ 296.184932] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 296.188480] LustreError: 15762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 296.363169] Lustre: server umount lustre-MDT0001 complete [ 299.330946] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 13:59:52 (1767034792) [ 309.381742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 309.673434] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.718357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 309.721405] Lustre: Skipped 1 previous similar message [ 312.156240] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 314.852403] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.990733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 319.368184] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 322.407626] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 324.224654] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 325.439725] Lustre: Failing over lustre-MDT0000 [ 325.449424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.450268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.454361] Lustre: Skipped 1 previous similar message [ 325.464057] Lustre: Skipped 5 previous similar messages [ 325.552257] LustreError: 20799:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 325.556362] LustreError: 20799:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 325.601430] Lustre: server umount lustre-MDT0000 complete [ 327.523319] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.531521] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 332.772575] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.786391] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 341.030727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 341.104763] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.242711] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 342.244928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 342.254632] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 342.951582] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 347.280637] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 352.740970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.741190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 352.746912] Lustre: Skipped 1 previous similar message [ 356.243102] Lustre: server umount lustre-MDT0000 complete [ 357.866257] LustreError: 19451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.866257] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.866270] LustreError: 19451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 357.872305] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 358.706861] LustreError: 19438:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767034851 with bad export cookie 4803680301519256396 [ 358.710987] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 358.717733] LustreError: 19438:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 358.823827] LustreError: 21982:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 358.846479] LustreError: 21982:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 358.958828] Lustre: server umount lustre-MDT0001 complete [ 364.530092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.851978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.855995] Lustre: Skipped 2 previous similar messages [ 367.292470] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 372.913161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 375.357459] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 378.851164] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 381.272252] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 382.645877] Lustre: Failing over lustre-MDT0000 [ 382.782347] Lustre: server umount lustre-MDT0000 complete [ 383.460580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 394.721318] LustreError: 22452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 394.731756] LustreError: 22452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 398.817745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.903204] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 400.106019] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 400.112446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 400.115985] Lustre: Skipped 1 previous similar message [ 400.129325] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 401.629148] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 406.682793] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 410.594038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 410.598532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.605245] Lustre: Skipped 2 previous similar messages [ 410.612257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 410.615953] Lustre: Skipped 2 previous similar messages [ 417.806913] Lustre: server umount lustre-MDT0000 complete [ 420.155720] LustreError: 22439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767034913 with bad export cookie 4803680301519257327 [ 420.158195] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 420.164196] LustreError: 22439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 420.352827] Lustre: server umount lustre-MDT0001 complete [ 425.580193] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:01:58 (1767034918) [ 435.616662] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 441.737715] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 448.274300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 452.416787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 455.864120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 459.785961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 465.425167] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 472.256636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 472.297269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.428648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 472.451918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 472.455887] Lustre: Skipped 1 previous similar message [ 472.500718] Lustre: lustre-MDT0000: new disk, initializing [ 472.503334] Lustre: Skipped 1 previous similar message [ 472.582253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.585451] Lustre: Skipped 2 previous similar messages [ 472.601556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 472.610530] Lustre: Skipped 1 previous similar message [ 474.692626] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 480.494345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 480.524972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 480.569941] Lustre: 28461:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 480.730187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 482.813309] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 487.488236] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 489.528567] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 493.682778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 493.735350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 495.253366] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 495.310824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 495.320988] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 497.376090] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 502.048566] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 507.114042] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 519.516094] LustreError: 31098:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 519.518895] LustreError: 31098:0:(obd_class.h:479:obd_check_dev()) Skipped 27 previous similar messages [ 519.593891] Lustre: server umount lustre-OST0000 complete [ 521.697243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 521.702080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.710756] Lustre: Skipped 1 previous similar message [ 521.713948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 521.717445] Lustre: Skipped 2 previous similar messages [ 531.486251] Lustre: server umount lustre-MDT0000 complete [ 533.612208] LustreError: 29242:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035026 with bad export cookie 4803680301519257726 [ 533.617405] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.617547] LustreError: 29242:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 543.323739] Lustre: server umount lustre-MDT0001 complete [ 549.720289] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 551.476394] Key type lgssc unregistered [ 551.733668] LNet: 32033:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.744296] LNetError: 32033:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 551.758712] LNet: Removed LNI 192.168.204.113@tcp [ 552.287225] Key type .llcrypt unregistered [ 552.289204] Key type ._llcrypt unregistered [ 564.867554] Key type ._llcrypt registered [ 564.869168] Key type .llcrypt registered [ 564.923304] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 565.368894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 565.436773] alg: No test for adler32 (adler32-zlib) [ 566.344907] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 566.471213] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 568.088223] Key type lgssc registered [ 568.704156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 572.955119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 572.964921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.216066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 576.299554] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 581.186078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 581.365800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 583.196516] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 585.862294] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 587.592412] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 590.579564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 590.736924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 591.788392] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 593.039617] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 595.330173] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 606.212340] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:04:59 (1767035099) [ 606.971362] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 607.741458] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 610.630917] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:05:03 (1767035103) [ 614.542245] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:05:07 (1767035107) [ 614.973618] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 618.923766] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 618.929603] Lustre: Skipped 2 previous similar messages [ 619.742759] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:05:12 (1767035112) [ 623.165988] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 623.169469] Lustre: Skipped 4 previous similar messages [ 630.101760] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:05:22 (1767035122) [ 631.433979] Lustre: *** cfs_fail_loc=131b, val=2*** [ 633.021203] Lustre: *** cfs_fail_loc=131b, val=2*** [ 637.214308] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:05:29 (1767035129) [ 638.124332] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 638.127134] Lustre: Skipped 9 previous similar messages [ 644.755559] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:05:37 (1767035137) [ 651.235190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 651.238847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 651.241294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 651.246406] Lustre: Skipped 1 previous similar message [ 656.356610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 656.359548] Lustre: Skipped 2 previous similar messages [ 656.629100] LustreError: 39957:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 656.704209] Lustre: server umount lustre-OST0000 complete [ 661.475042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.482172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 661.485057] Lustre: Skipped 1 previous similar message [ 664.590915] LustreError: 40160:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 664.594149] LustreError: 40160:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 664.671021] Lustre: server umount lustre-MDT0000 complete [ 666.358381] LustreError: 33385:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035159 with bad export cookie 13098876048048578040 [ 666.362559] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 666.367153] LustreError: 33385:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 666.448449] LustreError: 40361:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 666.451512] LustreError: 40361:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 666.518643] Lustre: server umount lustre-MDT0001 complete [ 670.658379] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 671.846937] Key type lgssc unregistered [ 672.001535] LNet: 40892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.005059] LNetError: 40892:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.014593] LNet: Removed LNI 192.168.204.113@tcp [ 672.359155] Key type .llcrypt unregistered [ 672.360606] Key type ._llcrypt unregistered [ 679.321515] Key type ._llcrypt registered [ 679.323170] Key type .llcrypt registered [ 679.377399] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 687.158874] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 687.649263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 687.660613] alg: No test for adler32 (adler32-zlib) [ 688.562378] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 688.669237] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 690.264432] Key type lgssc registered [ 690.856761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 694.971147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 697.912425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 700.780249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 703.758523] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 707.028298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 707.061022] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 707.067460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 708.184686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 708.202533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 708.244303] Lustre: lustre-MDT0000: new disk, initializing [ 708.297482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 708.310798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 709.975542] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 714.874680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 714.903174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 714.948328] Lustre: 43996:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 714.978745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 714.980739] Lustre: Skipped 1 previous similar message [ 715.020392] Lustre: lustre-MDT0001: new disk, initializing [ 715.063033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 715.079839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 715.084445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 716.638967] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 720.414670] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 721.948248] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 725.077310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 725.115513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 725.261456] Lustre: lustre-OST0000: new disk, initializing [ 725.264883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 725.312796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 727.245941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 727.250814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 727.314233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 727.936364] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 731.636201] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 733.280767] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 733.366039] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 735.033820] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 735.121299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 737.765877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 737.768237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.769189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 737.780609] Lustre: Skipped 1 previous similar message [ 742.067185] LustreError: 46934:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 742.135343] Lustre: server umount lustre-OST0000 complete [ 745.953591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 745.957491] LustreError: Skipped 1 previous similar message [ 745.960129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 745.966590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 745.969696] Lustre: Skipped 1 previous similar message [ 748.001535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 748.001931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 753.122746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 753.125771] Lustre: Skipped 2 previous similar messages [ 757.217949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 757.220796] Lustre: Skipped 1 previous similar message [ 758.240170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 758.281496] LustreError: 47135:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 758.283464] LustreError: 47135:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 758.312711] Lustre: server umount lustre-MDT0000 complete [ 759.854513] LustreError: 44775:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035252 with bad export cookie 5263520728263295105 [ 759.858217] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 759.859802] LustreError: 44775:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 759.931336] LustreError: 47338:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 759.934041] LustreError: 47338:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 759.998194] Lustre: server umount lustre-MDT0001 complete [ 764.216245] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:07:36 (1767035256) [ 767.163876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 767.400118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 769.059419] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 772.578263] LustreError: 48057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 772.586810] LustreError: 48057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 772.963485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 773.169278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 774.947151] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 777.801962] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 779.745603] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 783.191348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 783.351094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 785.866429] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 788.280765] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 795.979852] LustreError: 50251:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 796.419762] Lustre: Modifying parameter general.timeout=1 in log params [ 797.622741] Lustre: Modifying parameter general.timeout=4 in log params [ 797.627735] Lustre: Skipped 2 previous similar messages [ 799.721706] Lustre: Modifying parameter general.timeout=9 in log params [ 799.724308] Lustre: Skipped 4 previous similar messages [ 803.747911] Lustre: Modifying parameter general.timeout=20 in log params [ 803.750856] Lustre: Skipped 10 previous similar messages [ 811.803660] Lustre: Modifying parameter general.timeout=42 in log params [ 811.805377] Lustre: Skipped 21 previous similar messages [ 827.828700] Lustre: Modifying parameter general.timeout=83 in log params [ 827.831404] Lustre: Skipped 40 previous similar messages [ 848.916537] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:09:01 (1767035341) [ 849.928937] Lustre: 57155:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 851.243975] Lustre: 57352:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 851.249834] Lustre: 57352:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 853.386852] Lustre: 57647:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 853.392505] Lustre: 57647:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 860.214883] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 860.218712] Lustre: Skipped 65 previous similar messages [ 863.335582] Lustre: 58988:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 863.344081] LustreError: 58988:0:(mgs_handler.c:1129:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 864.906791] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:09:17 (1767035357) [ 870.368848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 870.371608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.376379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 875.456244] LustreError: 59681:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 875.458823] LustreError: 59681:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 875.488564] Lustre: server umount lustre-MDT0000 complete [ 876.001657] LustreError: 48056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 876.003052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 876.007096] LustreError: 48056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 876.011911] Lustre: Skipped 1 previous similar message [ 876.892471] LustreError: 50301:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035369 with bad export cookie 5263520728263295973 [ 876.895688] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 876.897531] LustreError: 50301:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 877.007615] Lustre: server umount lustre-MDT0001 complete [ 888.628081] LustreError: 60086:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 888.630131] LustreError: 60086:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 888.677969] Lustre: server umount lustre-OST0000 complete [ 891.936316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 893.474336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 894.977407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 896.467804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 899.153201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.264287] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 899.370523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.897377] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 904.236469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 904.275559] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 904.288269] Lustre: 61534:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 905.932233] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 907.999145] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 909.495740] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 912.294709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 912.379473] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 914.785588] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 916.903844] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 924.129397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 924.131638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.131857] LustreError: Skipped 1 previous similar message [ 924.135436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 924.138332] Lustre: Skipped 1 previous similar message [ 925.741133] LustreError: 64013:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 925.744364] LustreError: 64013:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 925.785664] Lustre: server umount lustre-OST0000 complete [ 933.269367] Lustre: server umount lustre-MDT0000 complete [ 934.369281] LustreError: 62269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.374611] LustreError: 62269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 934.653258] LustreError: 63175:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035427 with bad export cookie 5263520728263298360 [ 934.656958] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 934.658759] LustreError: 63175:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 934.785537] Lustre: server umount lustre-MDT0001 complete [ 938.839955] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 940.001791] Key type lgssc unregistered [ 940.142473] LNet: 64953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.146361] LNetError: 64953:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.153335] LNet: Removed LNI 192.168.204.113@tcp [ 940.499668] Key type .llcrypt unregistered [ 940.501356] Key type ._llcrypt unregistered [ 951.146907] Key type ._llcrypt registered [ 951.148351] Key type .llcrypt registered [ 951.188762] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 951.683658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 951.691353] alg: No test for adler32 (adler32-zlib) [ 952.556517] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 952.647430] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 954.240165] Key type lgssc registered [ 954.651985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 957.537050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 957.544579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.703093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.113768] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 963.264042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 963.453656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 964.837861] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 966.758916] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 967.995853] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 970.398393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 970.593712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.606341] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 974.482370] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 987.877336] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 994.276575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.276851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 994.277911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.281364] Lustre: Skipped 1 previous similar message [ 996.674343] LustreError: 69072:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 996.709860] Lustre: server umount lustre-MDT0000 complete [ 997.985735] LustreError: 68986:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035490 with bad export cookie 11544205524292871386 [ 997.989131] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 997.990376] LustreError: 68986:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 998.058923] LustreError: 69276:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 998.061991] LustreError: 69276:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 998.110255] Lustre: server umount lustre-MDT0001 complete [ 1009.715220] LustreError: 69476:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1009.717806] LustreError: 69476:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1009.760282] Lustre: server umount lustre-OST0000 complete [ 1013.005260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1014.417276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1015.905891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1017.355932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1019.886029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1019.987596] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1019.998658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1020.096553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.500939] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1024.682584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1024.724909] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1024.737093] Lustre: 70926:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1024.923164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1026.414481] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1028.374347] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1029.760293] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1032.403436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1032.472737] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1034.744917] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1036.688892] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1037.333575] Lustre: 73168:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1038.089394] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1044.449713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1044.451963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.453103] LustreError: Skipped 1 previous similar message [ 1044.453805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1044.458340] Lustre: Skipped 2 previous similar messages [ 1044.460408] Lustre: Skipped 2 previous similar messages [ 1045.612763] LustreError: 73417:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1045.615360] LustreError: 73417:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1045.656596] Lustre: server umount lustre-OST0000 complete [ 1049.570392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.571623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1049.575656] Lustre: Skipped 1 previous similar message [ 1054.689557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1054.692424] Lustre: Skipped 2 previous similar messages [ 1059.808967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1059.811384] Lustre: Skipped 1 previous similar message [ 1061.344126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1061.381649] LustreError: 73624:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1061.384230] LustreError: 73624:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1061.411932] Lustre: server umount lustre-MDT0000 complete [ 1062.783263] LustreError: 70964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035555 with bad export cookie 11544205524292872205 [ 1062.785918] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.788589] LustreError: 70964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1062.913507] Lustre: server umount lustre-MDT0001 complete [ 1066.780814] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1067.904736] Key type lgssc unregistered [ 1068.041367] LNet: 74358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.044791] LNetError: 74358:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.058526] LNet: Removed LNI 192.168.204.113@tcp [ 1068.385343] Key type .llcrypt unregistered [ 1068.387096] Key type ._llcrypt unregistered [ 1078.487344] Key type ._llcrypt registered [ 1078.488579] Key type .llcrypt registered [ 1078.530565] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1078.929956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1078.945887] alg: No test for adler32 (adler32-zlib) [ 1079.808731] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1079.899046] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1081.488246] Key type lgssc registered [ 1081.905869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1084.747467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1084.754451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.931177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.347794] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1090.586868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1090.798478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1092.317639] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1094.439069] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1095.789350] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1098.329656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1098.539994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.654839] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1102.655839] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1115.176225] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1115.850066] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1115.852863] Lustre: Skipped 1 previous similar message [ 1116.854756] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1116.857717] Lustre: Skipped 2 previous similar messages [ 1119.070143] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1119.072450] Lustre: Skipped 6 previous similar messages [ 1123.260128] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1123.263383] Lustre: Skipped 12 previous similar messages [ 1131.566202] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1131.568689] Lustre: Skipped 24 previous similar messages [ 1147.648515] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1147.651711] Lustre: Skipped 48 previous similar messages [ 1150.870635] Lustre: 83484:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1151.858262] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 14:14:04 (1767035644) [ 1152.479625] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1153.202937] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 14:14:06 (1767035646) [ 1157.091455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1157.091660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.093080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1157.094870] LustreError: Skipped 1 previous similar message [ 1157.098595] Lustre: Skipped 1 previous similar message [ 1160.368124] LustreError: 83776:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1160.407418] Lustre: server umount lustre-OST0000 complete [ 1162.209802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.210498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1162.214831] Lustre: Skipped 1 previous similar message [ 1162.217213] Lustre: Skipped 1 previous similar message [ 1167.329635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1167.331968] Lustre: Skipped 2 previous similar messages [ 1167.935451] LustreError: 83979:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1167.938145] LustreError: 83979:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1167.970103] Lustre: server umount lustre-MDT0000 complete [ 1169.403317] LustreError: 75709:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035662 with bad export cookie 2361390387534823770 [ 1169.406780] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.408075] LustreError: 75709:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1169.480817] LustreError: 84182:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1169.484100] LustreError: 84182:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1169.542237] Lustre: server umount lustre-MDT0001 complete [ 1173.625021] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1174.838069] Key type lgssc unregistered [ 1174.989430] LNet: 84712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1174.992801] LNetError: 84712:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.000360] LNet: Removed LNI 192.168.204.113@tcp [ 1175.341128] Key type .llcrypt unregistered [ 1175.342675] Key type ._llcrypt unregistered [ 1176.886979] Key type ._llcrypt registered [ 1176.888416] Key type .llcrypt registered [ 1176.926775] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1178.461176] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules [ 1178.873067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1178.889704] alg: No test for adler32 (adler32-zlib) [ 1179.754589] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1179.844115] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1181.424307] Key type lgssc registered [ 1181.865801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1182.333119] LustreError: 85828:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1190.697798] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1194.150367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1203.585568] LustreError: 85828:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1203.596556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1204.791591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1206.167736] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1209.280582] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 14:15:02 (1767035702) [ 1211.019371] LustreError: 86866:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1211.023818] LustreError: 86866:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1211.027627] LustreError: 86866:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1211.100079] LustreError: 87637:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1211.161331] Lustre: server umount lustre-MDT0000 complete [ 1216.132426] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1217.269149] Key type lgssc unregistered [ 1217.411365] LNet: 88322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1217.414174] LNetError: 88322:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1217.424366] LNet: Removed LNI 192.168.204.113@tcp [ 1217.744209] Key type .llcrypt unregistered [ 1217.745333] Key type ._llcrypt unregistered [ 1225.295040] Key type ._llcrypt registered [ 1225.296134] Key type .llcrypt registered [ 1225.333687] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1225.691176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1225.725518] alg: No test for adler32 (adler32-zlib) [ 1226.584287] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1226.668838] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1228.256154] Key type lgssc registered [ 1228.639799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1231.295605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1231.301762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.471917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1233.815973] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1236.773572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1236.994769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1238.366873] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1240.262489] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1241.508399] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1243.922369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1244.148426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.209533] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1248.107202] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1280.211172] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 14:16:13 (1767035773) [ 1280.792836] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1281.408406] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 14:16:14 (1767035774) [ 1283.041416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1283.044659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.049200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1284.066911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1284.068267] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.069930] Lustre: Skipped 1 previous similar message [ 1284.073746] Lustre: Skipped 1 previous similar message [ 1288.510760] LustreError: 92430:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1288.538686] Lustre: server umount lustre-MDT0000 complete [ 1289.185264] LustreError: 89498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.190610] LustreError: 89498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1289.760063] LustreError: 89478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035782 with bad export cookie 899254082440388135 [ 1289.762596] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1289.764526] LustreError: 89478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1289.830873] LustreError: 92631:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1289.833267] LustreError: 92631:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1289.870917] Lustre: server umount lustre-MDT0001 complete [ 1301.361717] LustreError: 92833:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1301.364744] LustreError: 92833:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1301.474507] Lustre: server umount lustre-OST0000 complete [ 1305.269384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1305.418839] LustreError: 93462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1305.476113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.745042] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1309.537189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1309.770947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1311.037270] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1312.969526] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1314.221313] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1316.300438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1318.612318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1318.635452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1318.703965] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1318.708331] LustreError: 93450:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1318.718608] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1318.722178] LustreError: 95324:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1318.724193] LustreError: 95324:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1318.725863] LustreError: 95324:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1318.751684] Lustre: server umount lustre-OST0000 complete [ 1318.753773] LustreError: 95324:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1321.565664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1321.637924] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1321.640584] LustreError: 93450:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1321.650700] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1321.653427] LustreError: 95789:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1321.655443] LustreError: 95789:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1321.658071] LustreError: 95789:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1321.679835] Lustre: server umount lustre-OST0000 complete [ 1321.681484] LustreError: 95789:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1322.629022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1324.806451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1324.867549] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1324.869940] Lustre: Found index 0 for lustre-OST0000, updating log [ 1324.873295] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1324.898586] Lustre: lustre-OST0000: new disk, initializing [ 1324.900246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1324.993124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.897707] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1328.147441] Lustre: Failing over lustre-OST0000 [ 1328.164564] LustreError: 96862:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1328.166880] LustreError: 96862:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1328.198713] Lustre: server umount lustre-OST0000 complete [ 1330.145932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1330.145969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1330.146541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1330.146545] Lustre: Skipped 1 previous similar message [ 1330.156688] Lustre: Skipped 1 previous similar message [ 1334.241190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1334.242898] Lustre: Skipped 2 previous similar messages [ 1335.548764] LustreError: 97062:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1335.550615] LustreError: 97062:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1335.578489] Lustre: server umount lustre-MDT0000 complete [ 1336.914385] LustreError: 93441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035829 with bad export cookie 899254082440390865 [ 1336.917277] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1336.918329] LustreError: 93441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1337.030634] Lustre: server umount lustre-MDT0001 complete [ 1339.937464] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 14:17:12 (1767035832) [ 1345.207396] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1348.722872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.852408] LustreError: 98633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.877506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.204648] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1353.143415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1354.664356] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1355.654440] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1358.071257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1360.431393] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1363.426484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1363.450735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1363.478195] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1363.522891] Lustre: lustre-OST0001: new disk, initializing [ 1363.525039] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1364.225902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1364.229110] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1364.277521] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1365.693338] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1369.142698] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1375.066040] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1382.296821] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.13@tcp (stopping) [ 1383.904779] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1383.907063] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1387.648815] LustreError: 101872:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1387.650696] LustreError: 101872:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1387.674446] Lustre: server umount lustre-MDT0001 complete [ 1388.604360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1390.052960] LustreError: 99347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.058869] LustreError: 99347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1390.767674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1390.801793] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1390.803741] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1390.819799] Lustre: 98622:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1391.065083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1391.067493] Lustre: Skipped 3 previous similar messages [ 1392.422941] LustreError: 102629:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1392.426650] LustreError: 102629:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1392.429752] LustreError: 98624:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1392.460645] LustreError: 102647:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.204.113@tcp: cfg command failed: rc = -17 [ 1392.466147] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1392.466147] [ 1392.475465] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1395.702909] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 14:18:08 (1767035888) [ 1396.202682] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1401.618504] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1407.416943] Lustre: 104477:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1415.397518] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1416.670463] Lustre: 105608:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1472.993053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1472.993710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.994491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1472.994496] Lustre: Skipped 5 previous similar messages [ 1473.004256] Lustre: Skipped 3 previous similar messages [ 1474.954285] LustreError: 106040:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1474.956814] LustreError: 106040:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1474.979487] Lustre: server umount lustre-MDT0000 complete [ 1476.261862] LustreError: 98614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767035969 with bad export cookie 899254082440391733 [ 1476.265701] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1476.266598] LustreError: 98614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1476.398802] Lustre: server umount lustre-MDT0001 complete [ 1487.309769] Lustre: server umount lustre-OST0000 complete [ 1498.784611] Lustre: server umount lustre-OST0001 complete [ 1500.803226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1503.683585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1503.989293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1506.906954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1509.723324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1510.049494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1516.401181] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1519.751838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1519.756854] Lustre: lustre-MDT0000: reset Object Index mappings [ 1519.912594] LustreError: 109518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1519.917510] LustreError: 109518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1519.936855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.202764] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1524.123729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1524.130732] Lustre: lustre-MDT0001: reset Object Index mappings [ 1525.766506] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1526.775186] Lustre: 110632:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1528.994921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1530.275430] LustreError: 110988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1530.280061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 1531.119703] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1533.983406] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1535.216557] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 1535.217298] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 1535.257421] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 1535.970771] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1539.131113] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1540.372783] Lustre: 112482:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1562.605121] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:20:55 (1767036055) [ 1591.778119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1591.778912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1591.783859] Lustre: Skipped 4 previous similar messages [ 1591.787381] Lustre: Skipped 6 previous similar messages [ 1597.582353] LustreError: 112834:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1597.585208] LustreError: 112834:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1597.620358] Lustre: server umount lustre-MDT0000 complete [ 1599.009450] LustreError: 109499:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036091 with bad export cookie 899254082440438479 [ 1599.011442] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1599.014564] LustreError: 109499:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1602.017180] LustreError: 111011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1602.023390] LustreError: 111011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1605.318264] Lustre: server umount lustre-MDT0001 complete [ 1612.817628] Lustre: server umount lustre-OST0000 complete [ 1620.264512] Lustre: server umount lustre-OST0001 complete [ 1622.475057] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 1625.829672] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1629.308387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1631.545881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1633.753238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1635.874139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1638.142850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1638.163975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.242667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1638.253262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1638.284524] Lustre: lustre-MDT0000: new disk, initializing [ 1638.309724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1639.570818] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1642.507696] Lustre: server umount lustre-MDT0000 complete [ 1643.405041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1645.552931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1645.577369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.693667] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1645.698900] Lustre: 116703:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1645.703678] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1647.146483] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1648.961150] Lustre: server umount lustre-MDT0000 complete [ 1651.844561] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 14:22:24 (1767036144) [ 1652.356930] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1652.956131] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 14:22:25 (1767036145) [ 1658.962279] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1662.594889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.748280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.751410] Lustre: Skipped 5 previous similar messages [ 1664.160236] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1667.063039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1667.086141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1667.118380] Lustre: 118720:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1667.142288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1667.144256] Lustre: Skipped 1 previous similar message [ 1667.182343] Lustre: lustre-MDT0001: new disk, initializing [ 1667.226416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1667.229327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1668.529116] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1670.810109] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1673.054693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1673.077906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1673.179055] Lustre: lustre-OST0000: new disk, initializing [ 1673.181243] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1675.064906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1675.068658] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1675.105394] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1675.111205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1679.428491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1679.459667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1681.034740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1681.040155] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1681.072626] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1681.490196] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1685.900347] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1687.086643] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1839.541947] Lustre: 127487:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1839.545636] Lustre: 127487:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1839.548193] Lustre: 127487:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1839.550922] Lustre: 127487:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1839.552720] Lustre: 127487:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1839.554368] Lustre: 127487:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1915.848964] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:26:48 (1767036408) [ 1928.672851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1928.675234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1928.678593] Lustre: Skipped 3 previous similar messages [ 1928.680436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1928.681822] Lustre: Skipped 8 previous similar messages [ 1933.569902] LustreError: 127695:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1933.571624] LustreError: 127695:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 1933.622931] Lustre: server umount lustre-MDT0000 complete [ 1934.807203] LustreError: 127519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036427 with bad export cookie 899254082440493338 [ 1934.810637] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.812471] LustreError: 127519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1937.377139] LustreError: 121914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1937.382939] LustreError: 121914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1949.152095] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1949.271773] Lustre: server umount lustre-MDT0001 complete [ 1956.674613] Lustre: server umount lustre-OST0000 complete [ 1964.119183] Lustre: server umount lustre-OST0001 complete [ 1966.087428] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 1968.994777] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1972.068500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1974.118055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1976.061872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1978.319818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1980.772311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1980.798874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1980.898305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1980.912340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1980.915264] Lustre: Skipped 1 previous similar message [ 1980.955613] Lustre: lustre-MDT0000: new disk, initializing [ 1980.957120] Lustre: Skipped 1 previous similar message [ 1980.981430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.983398] Lustre: Skipped 3 previous similar messages [ 1980.989311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1982.311316] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1986.636672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1986.666016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1986.705997] Lustre: 130482:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1986.831245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1988.334152] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1991.561622] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1992.805850] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1995.267113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1995.293813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1996.992332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1996.996832] Lustre: Skipped 1 previous similar message [ 1996.999283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1997.054647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1997.592228] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2000.759917] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2004.021099] Lustre: lustre-MDD0000: changelog on [ 2004.955064] Lustre: lustre-MDD0001: changelog on [ 2343.946281] Lustre: lustre-MDD0001: changelog off [ 2344.548708] Lustre: 130489:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2344.834276] Lustre: lustre-MDD0000: changelog off [ 2350.562458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2350.563616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.564414] LustreError: Skipped 1 previous similar message [ 2350.569194] Lustre: Skipped 5 previous similar messages [ 2350.570852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2350.574051] Lustre: Skipped 11 previous similar messages [ 2352.365120] LustreError: 139590:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2352.367081] LustreError: 139590:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2352.408858] Lustre: server umount lustre-OST0000 complete [ 2359.695438] Lustre: server umount lustre-MDT0000 complete [ 2360.802262] LustreError: 130492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.807028] LustreError: 130492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2360.941615] LustreError: 130474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036853 with bad export cookie 899254082441259663 [ 2360.945739] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2360.947043] LustreError: 130474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2361.062804] Lustre: server umount lustre-MDT0001 complete [ 2364.721691] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2365.757378] Key type lgssc unregistered [ 2365.888353] LNet: 140523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2365.891278] LNetError: 140523:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2365.901419] LNet: Removed LNI 192.168.204.113@tcp [ 2366.218120] Key type .llcrypt unregistered [ 2366.219813] Key type ._llcrypt unregistered [ 2372.821289] Key type ._llcrypt registered [ 2372.822927] Key type .llcrypt registered [ 2372.866820] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 2379.061188] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2379.347526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2379.357458] alg: No test for adler32 (adler32-zlib) [ 2380.215949] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2380.309865] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2381.896200] Key type lgssc registered [ 2382.282363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2385.359817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2387.603481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2389.662064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2391.867886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2394.238789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2394.264276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2394.270363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.374380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2395.387818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2395.423197] Lustre: lustre-MDT0000: new disk, initializing [ 2395.451717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2395.458985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2396.913325] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2401.139273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2401.159414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2401.198690] Lustre: 143670:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2401.227380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2401.229115] Lustre: Skipped 1 previous similar message [ 2401.263384] Lustre: lustre-MDT0001: new disk, initializing [ 2401.298079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2401.312886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2401.316693] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2402.805615] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2406.059949] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2407.330768] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2409.778597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2409.801176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2409.893229] Lustre: lustre-OST0000: new disk, initializing [ 2409.895082] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2409.922172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.011465] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2412.605696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2412.609593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2412.652522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2415.178195] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2417.392596] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_module obdecho/obdecho [ 2418.037679] LustreError: 146293:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2418.040211] LustreError: 146293:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 2418.618643] LustreError: 146389:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2422.756853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.759107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2422.761613] Lustre: Skipped 2 previous similar messages [ 2422.765152] Lustre: Skipped 1 previous similar message [ 2427.876720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2427.879823] Lustre: Skipped 2 previous similar messages [ 2427.912070] LustreError: 146728:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2427.948212] Lustre: server umount lustre-MDT0000 complete [ 2429.182321] LustreError: 144450:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036922 with bad export cookie 6915639002294965747 [ 2429.185413] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2429.186601] LustreError: 144450:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2429.250085] LustreError: 146929:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2429.252043] LustreError: 146929:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2429.298164] Lustre: server umount lustre-MDT0001 complete [ 2440.818532] LustreError: 147131:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2440.820272] LustreError: 147131:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2440.860188] Lustre: server umount lustre-OST0000 complete [ 2447.992676] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 2451.356383] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2454.910203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2457.209296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2459.410550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2461.675881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2464.098194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2464.119495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2464.202366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2464.213944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2464.250109] Lustre: lustre-MDT0000: new disk, initializing [ 2464.274773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2464.281471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2465.679408] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2469.851565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2469.871172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2469.902309] Lustre: 150071:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2469.928548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2469.931566] Lustre: Skipped 1 previous similar message [ 2469.969457] Lustre: lustre-MDT0001: new disk, initializing [ 2469.999296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2470.009115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2470.012457] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2471.263470] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2474.395669] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2475.653904] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2478.002990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2478.023309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2478.115776] Lustre: lustre-OST0000: new disk, initializing [ 2478.117600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2478.145077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2479.419117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2479.424801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2479.466494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2480.136059] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2483.308531] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2484.745044] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2484.827345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2486.155432] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2486.224139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2488.943232] LustreError: 153008:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2488.946945] LustreError: 153008:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2488.988437] Lustre: server umount lustre-OST0000 complete [ 2490.848996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2490.852760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.856317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2494.945753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2494.946154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2500.065804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2500.067507] Lustre: Skipped 2 previous similar messages [ 2500.070665] LustreError: 153209:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2500.073290] LustreError: 153209:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2500.100577] Lustre: server umount lustre-MDT0000 complete [ 2501.342219] LustreError: 150062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767036994 with bad export cookie 6915639002294966692 [ 2501.345310] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2501.346833] LustreError: 150062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2511.845486] LustreError: 153412:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2511.847420] LustreError: 153412:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2511.903150] Lustre: server umount lustre-MDT0001 complete [ 2513.236673] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:36:46 (1767037006) [ 2513.814663] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2514.450333] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:36:47 (1767037007) [ 2517.092906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.277434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2518.759471] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2521.720627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2523.292995] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2525.199933] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2526.516152] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2528.906285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2530.979871] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2532.835511] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2547.680969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2547.683375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.690619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2551.754189] LustreError: 156352:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2551.756517] LustreError: 156352:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2551.795271] Lustre: server umount lustre-MDT0000 complete [ 2553.170291] LustreError: 154801:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037046 with bad export cookie 6915639002294967560 [ 2553.174037] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2553.176062] LustreError: 154801:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2553.744990] LustreError: 154079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.754323] LustreError: 154079:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2553.826265] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.831760] Lustre: Skipped 2 previous similar messages [ 2558.864768] LustreError: 154078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.870560] LustreError: 154078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2559.360543] Lustre: server umount lustre-MDT0001 complete [ 2561.688961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2566.186783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2566.268203] LustreError: 157611:0:(llog.c:1643:llog_backup()) MGC192.168.204.113@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2566.271203] Lustre: 157611:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.113@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2584.464393] LustreError: 157619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.468681] LustreError: 157619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2589.584472] LustreError: 157635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.588447] LustreError: 157635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2594.704895] LustreError: 157618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.712207] LustreError: 157618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2604.944398] LustreError: 157618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.952304] LustreError: 157618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2622.433180] LustreError: 157618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2622.437301] LustreError: 157618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 2627.168217] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2627.170126] Lustre: 157611:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.204.113@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2655.120592] LustreError: 157637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.128205] LustreError: 157637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [ 2656.224279] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2656.226069] Lustre: 154075:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.204.113@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2668.010199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2668.012653] Lustre: Skipped 2 previous similar messages [ 2669.526792] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2672.646154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2673.125509] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2674.144874] LustreError: 141885:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000088245b1d x1852872456242560/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2675.723939] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2677.668635] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2678.287680] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2679.271135] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2679.575948] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2682.235117] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2687.384707] LustreError: 159096:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2687.387947] LustreError: 159096:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2687.422932] Lustre: server umount lustre-MDT0000 complete [ 2688.752320] LustreError: 154065:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037181 with bad export cookie 6915639002294974014 [ 2688.757332] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2688.887407] Lustre: server umount lustre-MDT0001 complete [ 2710.742144] Lustre: server umount lustre-OST0000 complete [ 2717.852689] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 2721.489985] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2725.322285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2727.617960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2729.676610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2731.742758] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2734.056321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2734.080996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.170656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2734.182049] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2734.219602] Lustre: lustre-MDT0000: new disk, initializing [ 2734.245459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.247364] Lustre: Skipped 1 previous similar message [ 2734.252597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2735.587886] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2739.836985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2739.856225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2739.890058] Lustre: 162440:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2739.995278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2741.435129] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2744.685856] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2746.028255] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2748.429606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2748.458833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2750.648741] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2753.876109] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2755.436800] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2757.630139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2757.669905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2758.573634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2759.973728] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2760.053493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2762.723803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.730092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2762.730356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2762.736604] Lustre: Skipped 9 previous similar messages [ 2762.799106] LustreError: 165402:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2762.801175] LustreError: 165402:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2762.841690] Lustre: server umount lustre-OST0000 complete [ 2765.792937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2765.795484] LustreError: Skipped 1 previous similar message [ 2770.338338] Lustre: server umount lustre-MDT0000 complete [ 2771.711612] LustreError: 162430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037264 with bad export cookie 6915639002294990800 [ 2771.715563] LustreError: 162430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2771.715784] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.856268] Lustre: server umount lustre-MDT0001 complete [ 2775.141671] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:41:07 (1767037267) [ 2777.532708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2777.689383] LustreError: 166475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.696357] LustreError: 166475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 2779.195877] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2782.346871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2783.749031] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2785.499493] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2786.768391] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2788.931264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2790.861708] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2792.625657] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2806.926815] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:41:39 (1767037299) [ 2812.898087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2812.898518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.901641] LustreError: Skipped 1 previous similar message [ 2812.905385] Lustre: Skipped 3 previous similar messages [ 2812.907981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2812.910240] Lustre: Skipped 4 previous similar messages [ 2822.112127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2822.164280] Lustre: server umount lustre-OST0000 complete [ 2823.649084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2829.587791] Lustre: server umount lustre-MDT0000 complete [ 2830.897269] LustreError: 168851:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037323 with bad export cookie 6915639002294991668 [ 2830.900739] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2830.903093] LustreError: 168851:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2831.025874] Lustre: server umount lustre-MDT0001 complete [ 2834.857810] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2835.861414] Key type lgssc unregistered [ 2835.987405] LNet: 169874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2835.989963] LNetError: 169874:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2835.998507] LNet: Removed LNI 192.168.204.113@tcp [ 2836.342459] Key type .llcrypt unregistered [ 2836.344239] Key type ._llcrypt unregistered [ 2844.551969] Key type ._llcrypt registered [ 2844.553047] Key type .llcrypt registered [ 2844.597351] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2845.092959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2845.133758] alg: No test for adler32 (adler32-zlib) [ 2846.011977] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2846.116225] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2847.704139] Key type lgssc registered [ 2848.094955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2850.842866] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2850.849402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2866.336397] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2871.456274] LustreError: 171074:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.113@tcp: failed processing log, type 4: rc = -110 [ 2876.576416] LustreError: 171074:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2876.580264] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2876.582294] LustreError: MGC192.168.204.113@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2876.585560] LustreError: 171074:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2876.588629] LustreError: 171074:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 2876.591441] LustreError: 171074:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2876.593980] LustreError: 171074:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2876.615399] Lustre: server umount lustre-OST0000 complete [ 2876.617635] LustreError: 171074:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2879.626331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.779111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.180154] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2885.191369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2885.329940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2886.756993] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2888.678548] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2889.983277] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2892.251793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2892.326110] LustreError: 173164:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2892.329879] Lustre: 173164:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.204.113@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2892.382555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2894.484908] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2896.342167] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2900.963728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2900.964612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2900.965127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2900.967319] Lustre: Skipped 2 previous similar messages [ 2903.228075] LustreError: 173815:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2903.229988] LustreError: 173815:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2903.256153] Lustre: server umount lustre-MDT0000 complete [ 2904.525312] LustreError: 171555:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037397 with bad export cookie 2576050242582802065 [ 2904.529222] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2904.530712] LustreError: 171555:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2904.601630] LustreError: 174020:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2904.605201] LustreError: 174020:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2904.657485] Lustre: server umount lustre-MDT0001 complete [ 2915.187208] LustreError: 174220:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2915.189602] LustreError: 174220:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2915.232247] Lustre: server umount lustre-OST0000 complete [ 2918.308865] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 2921.790331] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2925.173748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2927.410786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2929.518717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2931.784638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2934.206373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2934.228459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2934.316694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2934.327833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2934.365536] Lustre: lustre-MDT0000: new disk, initializing [ 2934.391263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.398313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2935.787436] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2940.159829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2940.188763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2940.228031] Lustre: 176552:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2940.252239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2940.254066] Lustre: Skipped 1 previous similar message [ 2940.290034] Lustre: lustre-MDT0001: new disk, initializing [ 2940.320672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2940.331220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2940.333956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2941.576231] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2944.538832] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2945.737523] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2947.849811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2947.872809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2947.955791] Lustre: lustre-OST0000: new disk, initializing [ 2947.957941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2949.732649] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2951.157607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2951.162158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2951.188388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2952.663170] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2954.029810] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2954.106281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2955.487135] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2955.563859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2956.259297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2956.259623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.260782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2956.260787] Lustre: Skipped 2 previous similar messages [ 2956.261362] LustreError: Skipped 1 previous similar message [ 2956.264904] Lustre: Skipped 1 previous similar message [ 2958.253176] LustreError: 179493:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2958.255091] LustreError: 179493:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2958.290777] Lustre: server umount lustre-OST0000 complete [ 2960.864966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2960.867219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.870785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.873589] Lustre: Skipped 1 previous similar message [ 2965.659590] Lustre: server umount lustre-MDT0000 complete [ 2966.497913] LustreError: 176562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.502867] LustreError: 176562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2966.919074] LustreError: 176544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037459 with bad export cookie 2576050242582802765 [ 2966.924368] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2967.003749] LustreError: 179895:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2967.005771] LustreError: 179895:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2967.068359] Lustre: server umount lustre-MDT0001 complete [ 2969.858130] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:44:22 (1767037462) [ 2975.794646] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2976.857509] Key type lgssc unregistered [ 2976.994555] LNet: 181079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2976.997292] LNetError: 181079:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2977.005589] LNet: Removed LNI 192.168.204.113@tcp [ 2977.351262] Key type .llcrypt unregistered [ 2977.352434] Key type ._llcrypt unregistered [ 2984.810731] Key type ._llcrypt registered [ 2984.812458] Key type .llcrypt registered [ 2984.849131] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2985.239625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2985.289803] alg: No test for adler32 (adler32-zlib) [ 2986.132971] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 2986.213706] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2987.792160] Key type lgssc registered [ 2988.154443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2990.867555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2990.873759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.029687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2993.404094] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2996.291132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2996.401314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2997.788106] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2999.581783] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3000.774860] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3003.383340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3003.453231] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3003.454538] LustreError: 183896:0:(llog.c:1684:llog_backup()) MGC192.168.204.113@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3003.457061] Lustre: 183896:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.204.113@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3003.498937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3005.464370] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3007.329708] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3008.034543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3008.037230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3008.038900] Lustre: Skipped 1 previous similar message [ 3013.089685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3013.091543] Lustre: Skipped 4 previous similar messages [ 3018.212132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3018.214231] Lustre: Skipped 2 previous similar messages [ 3022.304088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3022.337895] LustreError: 184551:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3022.361543] Lustre: server umount lustre-MDT0000 complete [ 3023.330197] LustreError: 182970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3023.335463] LustreError: 182970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3023.581309] LustreError: 182236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037516 with bad export cookie 2006096955400474948 [ 3023.584831] LustreError: 182236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3023.584930] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3023.651462] LustreError: 184754:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3023.653525] LustreError: 184754:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3023.697492] Lustre: server umount lustre-MDT0001 complete [ 3034.162130] LustreError: 184955:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3034.163995] LustreError: 184955:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3034.203758] Lustre: server umount lustre-OST0000 complete [ 3037.283980] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 3040.553745] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3043.821761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3046.073574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3048.244760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3050.361404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3052.658848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3052.684191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.772475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3052.784350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3052.830215] Lustre: lustre-MDT0000: new disk, initializing [ 3052.855768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3052.863033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3054.210498] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3058.361980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3058.386164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3058.415831] Lustre: 187286:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3058.432190] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3058.433245] Lustre: Skipped 1 previous similar message [ 3058.434297] LustreError: 188053:0:(llog.c:1684:llog_backup()) MGC192.168.204.113@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3058.436480] LustreError: 188053:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 3058.438137] Lustre: 188053:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.113@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3058.444797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3058.446381] Lustre: Skipped 1 previous similar message [ 3058.486132] Lustre: lustre-MDT0001: new disk, initializing [ 3058.524775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3058.535433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3058.539067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3059.888255] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3063.081030] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3064.319743] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3066.602825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3066.633904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3066.728402] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3066.729506] Lustre: Skipped 1 previous similar message [ 3066.730619] LustreError: 189009:0:(llog.c:1684:llog_backup()) MGC192.168.204.113@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3066.733305] LustreError: 189009:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 3066.735103] Lustre: 189009:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.113@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3066.737300] Lustre: 189009:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3066.746586] Lustre: lustre-OST0000: new disk, initializing [ 3066.748313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3066.776495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3066.822676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3066.826119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3066.867451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3068.646338] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3071.710526] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3073.136032] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3073.222768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3074.628110] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3074.712783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3077.090589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3077.091460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.093995] LustreError: Skipped 1 previous similar message [ 3077.097758] Lustre: Skipped 1 previous similar message [ 3077.099852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3077.102903] Lustre: Skipped 1 previous similar message [ 3077.425919] LustreError: 190226:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3077.427830] LustreError: 190226:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3077.461479] Lustre: server umount lustre-OST0000 complete [ 3079.136892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3079.140611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.210501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3082.211229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.212696] Lustre: Skipped 1 previous similar message [ 3084.804390] LustreError: 190427:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3084.806648] LustreError: 190427:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3084.840175] Lustre: server umount lustre-MDT0000 complete [ 3086.107471] LustreError: 188067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037579 with bad export cookie 2006096955400475648 [ 3086.110493] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3086.111667] LustreError: 188067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3086.235991] Lustre: server umount lustre-MDT0001 complete [ 3089.168275] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:46:22 (1767037582) [ 3094.619722] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3098.023764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3098.172108] LustreError: 191994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3098.186871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.454536] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3102.266697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3103.675062] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3104.623609] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3106.862433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3108.890840] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3111.831105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3111.856495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3111.921698] Lustre: lustre-OST0001: new disk, initializing [ 3111.924924] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3113.071443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3113.074831] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3113.089881] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3113.901439] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3119.384387] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3122.075418] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3127.720224] Lustre: DEBUG MARKER: ADD OST3 [ 3129.418524] loop: module loaded [ 3129.438148] loop0: detected capacity change from 0 to 204800000 [ 3129.526716] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3129.532445] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3129.538785] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3129.595948] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3131.679842] loop0: detected capacity change from 0 to 204800000 [ 3133.530440] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3133.561200] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3133.640191] Lustre: lustre-OST0003: new disk, initializing [ 3133.641917] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3133.688925] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3133.691236] Lustre: Skipped 3 previous similar messages [ 3134.659874] Lustre: DEBUG MARKER: STOP OST3 [ 3135.063925] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3135.111754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3135.117359] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3135.130092] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3135.369695] Lustre: Failing over lustre-OST0003 [ 3135.393229] LustreError: 196363:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3135.396317] LustreError: 196363:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3135.420575] Lustre: server umount lustre-OST0003 complete [ 3135.915943] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3137.121100] LustreError: 196271:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3139.935520] Lustre: DEBUG MARKER: START OST3 again [ 3140.510463] LustreError: 193448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.583025] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3140.584257] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.591826] Lustre: Skipped 1 previous similar message [ 3142.303314] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3142.402703] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3143.532736] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3144.318779] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3144.356564] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3144.356732] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3157.936474] Lustre: Failing over lustre-OST0003 [ 3157.959095] LustreError: 198287:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3157.960796] LustreError: 198287:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3157.976518] Lustre: server umount lustre-OST0003 complete [ 3157.992328] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3157.994626] LustreError: 195289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3157.996100] Lustre: Skipped 1 previous similar message [ 3158.004610] LustreError: 195289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3163.110253] LustreError: 193447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.113638] LustreError: 193447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3173.347963] LustreError: 193447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.349783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3173.353605] LustreError: 193447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3173.355477] Lustre: Skipped 3 previous similar messages [ 3176.542535] Lustre: server umount lustre-MDT0000 complete [ 3177.768382] LustreError: 191976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037670 with bad export cookie 2006096955400476516 [ 3177.771868] LustreError: 191976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3177.772387] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3178.466250] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3178.471770] Lustre: Skipped 4 previous similar messages [ 3192.288131] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3192.323278] LustreError: 198759:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 3192.325124] LustreError: 198759:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3192.380979] Lustre: server umount lustre-MDT0001 complete [ 3193.738755] Lustre: server umount lustre-OST0000 complete [ 3195.086858] Lustre: server umount lustre-OST0001 complete [ 3197.221532] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 3199.899848] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3203.472629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3205.829068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3208.009936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3210.339987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3212.694994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3212.714182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3212.796563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3212.807924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3212.846037] Lustre: lustre-MDT0000: new disk, initializing [ 3212.869743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.873034] Lustre: Skipped 1 previous similar message [ 3212.879942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3214.176114] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3218.367471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3218.393721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3218.429888] Lustre: 201345:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3218.550829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3219.914605] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3222.999782] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3224.122085] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3226.316354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3226.342802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3228.091461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3228.125066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3228.313817] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3231.478100] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3232.856691] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3232.926039] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3234.238875] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3234.304906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3237.011907] Lustre: server umount lustre-OST0000 complete [ 3238.369768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3238.370548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3238.373295] Lustre: Skipped 2 previous similar messages [ 3238.376757] Lustre: Skipped 7 previous similar messages [ 3248.641405] Lustre: server umount lustre-MDT0000 complete [ 3249.906426] LustreError: 203094:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037742 with bad export cookie 2006096955400478798 [ 3249.909697] LustreError: 203094:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3249.910050] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3259.365667] LustreError: 204689:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3259.367674] LustreError: 204689:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 3259.414487] Lustre: server umount lustre-MDT0001 complete [ 3260.643481] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:49:13 (1767037753) [ 3265.797712] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 3269.236964] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3272.369188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3274.649968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3276.737469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3278.935767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3281.198606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3281.224663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3281.314847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3281.326839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3281.328711] Lustre: Skipped 3 previous similar messages [ 3281.365293] Lustre: lustre-MDT0000: new disk, initializing [ 3281.367839] Lustre: Skipped 2 previous similar messages [ 3281.403625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3281.406772] Lustre: Skipped 2 previous similar messages [ 3282.738435] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3286.847392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3286.872514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3286.905942] Lustre: 207527:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3287.007466] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3288.338252] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3291.356881] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3292.589454] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3294.884198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3294.908712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3296.282795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3296.918496] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3300.081700] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3301.539272] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3301.622883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3302.979179] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3303.056874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3306.465832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3306.467719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3306.470144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3306.472836] Lustre: Skipped 1 previous similar message [ 3306.475786] Lustre: Skipped 5 previous similar messages [ 3309.779290] Lustre: server umount lustre-OST0000 complete [ 3317.083435] Lustre: server umount lustre-MDT0000 complete [ 3318.308077] LustreError: 208308:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037811 with bad export cookie 2006096955400479680 [ 3318.311210] LustreError: 208308:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3318.311248] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3318.428764] Lustre: server umount lustre-MDT0001 complete [ 3321.291714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.446370] LustreError: 211349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.453414] LustreError: 211349:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3322.825362] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3325.837540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3327.357630] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3329.138785] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3330.346461] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3332.577431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3334.553762] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3336.383069] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3358.692805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3358.695155] LustreError: Skipped 1 previous similar message [ 3364.371815] Lustre: server umount lustre-OST0000 complete [ 3366.880911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3371.743356] Lustre: server umount lustre-MDT0000 complete [ 3373.061520] LustreError: 211329:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767037866 with bad export cookie 2006096955400480562 [ 3373.066666] LustreError: 211329:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3373.067263] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3373.199715] Lustre: server umount lustre-MDT0001 complete [ 3468.180126] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 3469.229256] Key type lgssc unregistered [ 3469.357409] LNet: 214680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3469.360110] LNetError: 214680:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3469.367291] LNet: Removed LNI 192.168.204.113@tcp [ 3469.694118] Key type .llcrypt unregistered [ 3469.695299] Key type ._llcrypt unregistered [ 3477.977539] Key type ._llcrypt registered [ 3477.978896] Key type .llcrypt registered [ 3478.022712] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 3484.351076] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3484.736307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3484.745116] alg: No test for adler32 (adler32-zlib) [ 3485.619363] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 3485.722298] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3487.312156] Key type lgssc registered [ 3487.734041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3490.789184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3493.041789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3495.142033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3497.340245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3499.660532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3499.681467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3499.686590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.774979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3500.786695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3500.822663] Lustre: lustre-MDT0000: new disk, initializing [ 3500.846775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.853938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3502.175657] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3506.419574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3506.440211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3506.471450] Lustre: 217981:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3506.495579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3506.497481] Lustre: Skipped 1 previous similar message [ 3506.533190] Lustre: lustre-MDT0001: new disk, initializing [ 3506.565353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3506.576969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3506.580115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3507.893410] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3511.063401] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3512.297122] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3514.651424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3514.679150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3514.772811] Lustre: lustre-OST0000: new disk, initializing [ 3514.774851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3514.804757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3516.673872] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3517.753213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3517.756276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3517.787641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3519.782030] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3521.255080] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3521.334723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3522.706453] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3522.783419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3528.161529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3528.164164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3528.165028] LustreError: Skipped 1 previous similar message [ 3528.170061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3528.173137] Lustre: Skipped 1 previous similar message [ 3529.521100] LustreError: 220919:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3529.566768] Lustre: server umount lustre-OST0000 complete [ 3532.256911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3532.259150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.262919] Lustre: Skipped 1 previous similar message [ 3532.264625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3533.281755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3533.281835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.024523] LustreError: 221120:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3537.027065] LustreError: 221120:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3537.059101] Lustre: server umount lustre-MDT0000 complete [ 3538.335195] LustreError: 217971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038031 with bad export cookie 17764027511017164585 [ 3538.339681] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3538.341054] LustreError: 217971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3538.401640] LustreError: 221321:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3538.403836] LustreError: 221321:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3538.454944] Lustre: server umount lustre-MDT0001 complete [ 3544.801984] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3548.357211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3548.530338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3549.915850] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3552.897886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3553.040567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3554.440581] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3555.499073] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3557.916631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3560.097873] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3563.187939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3563.216145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3563.273372] Lustre: lustre-OST0001: new disk, initializing [ 3563.275452] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3563.313466] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3563.315253] Lustre: Skipped 1 previous similar message [ 3565.286382] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3567.625513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3567.629692] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3567.675113] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3570.745704] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3575.446469] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3584.226522] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:54:37 (1767038077) [ 3588.069120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.073519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3588.074231] Lustre: Skipped 2 previous similar messages [ 3588.077061] Lustre: Skipped 1 previous similar message [ 3591.047809] LustreError: 225871:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3591.051187] LustreError: 225871:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3591.094551] Lustre: server umount lustre-MDT0000 complete [ 3592.348311] LustreError: 222474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038085 with bad export cookie 17764027511017165453 [ 3592.351701] LustreError: 222474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3592.352044] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3592.467670] Lustre: server umount lustre-MDT0001 complete [ 3603.893163] LustreError: 226272:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3603.896550] LustreError: 226272:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3603.912136] Lustre: server umount lustre-OST0000 complete [ 3615.412963] LustreError: 226473:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3615.415081] LustreError: 226473:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3615.464925] Lustre: server umount lustre-OST0001 complete [ 3617.525430] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 3620.754423] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3624.239561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3626.484642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3628.681491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3630.789799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3633.144657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3633.169514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.262380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3633.276975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3633.326508] Lustre: lustre-MDT0000: new disk, initializing [ 3633.354957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.362365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3634.693066] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3638.939956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3638.963317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3638.995896] Lustre: 228658:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3639.109705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3640.462414] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3643.573885] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3644.848300] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3647.101035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3647.128564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3647.221415] Lustre: lustre-OST0000: new disk, initializing [ 3647.222912] Lustre: Skipped 1 previous similar message [ 3647.224643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3647.228023] Lustre: Skipped 2 previous similar messages [ 3648.505769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3648.509430] Lustre: Skipped 1 previous similar message [ 3648.511268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3648.547423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3649.205704] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3652.311846] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3653.707047] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3653.797695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3655.180337] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3655.248260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3658.722488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3658.723583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3658.725377] LustreError: Skipped 1 previous similar message [ 3658.729958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3658.732759] Lustre: Skipped 4 previous similar messages [ 3661.935227] LustreError: 231594:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3661.938390] LustreError: 231594:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3661.982695] Lustre: server umount lustre-OST0000 complete [ 3668.961328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3668.963516] Lustre: Skipped 3 previous similar messages [ 3669.464918] Lustre: server umount lustre-MDT0000 complete [ 3670.716233] LustreError: 229437:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038163 with bad export cookie 17764027511017167553 [ 3670.719662] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3670.720116] LustreError: 229437:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3670.855339] Lustre: server umount lustre-MDT0001 complete [ 3673.709282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3673.858188] LustreError: 232473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3673.863330] LustreError: 232473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3673.880272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3673.882299] Lustre: Skipped 2 previous similar messages [ 3675.247674] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3678.077365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3679.649180] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3681.445985] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3682.669502] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3684.952953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3686.979175] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3688.762213] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3836.901848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3836.902114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.904019] LustreError: Skipped 1 previous similar message [ 3836.907375] Lustre: Skipped 3 previous similar messages [ 3836.908414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3836.911521] Lustre: Skipped 1 previous similar message [ 3840.173667] LustreError: 234756:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3840.175991] LustreError: 234756:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3840.222172] Lustre: server umount lustre-OST0000 complete [ 3842.018127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3847.582328] Lustre: server umount lustre-MDT0000 complete [ 3848.854163] LustreError: 234090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038341 with bad export cookie 17764027511017168435 [ 3848.856933] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3848.857163] LustreError: 234090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3848.986321] Lustre: server umount lustre-MDT0001 complete [ 3951.158330] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 3952.320624] Key type lgssc unregistered [ 3952.471327] LNet: 235733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3952.473534] LNetError: 235733:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3952.483330] LNet: Removed LNI 192.168.204.113@tcp [ 3952.771103] Key type .llcrypt unregistered [ 3952.772151] Key type ._llcrypt unregistered [ 3960.855944] Key type ._llcrypt registered [ 3960.856939] Key type .llcrypt registered [ 3960.893972] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 3967.124593] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3967.455407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3967.463705] alg: No test for adler32 (adler32-zlib) [ 3968.314943] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 3968.412445] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3970.000170] Key type lgssc registered [ 3970.366031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3973.307956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3975.596559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3977.676600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3979.890898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3985.661628] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3989.373265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3989.391947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3989.396561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3990.486559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3990.500957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3990.544301] Lustre: lustre-MDT0000: new disk, initializing [ 3990.570234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3990.576609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3991.951969] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3996.151610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3996.176301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3996.208741] Lustre: 239740:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3996.233923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3996.235603] Lustre: Skipped 1 previous similar message [ 3996.274731] Lustre: lustre-MDT0001: new disk, initializing [ 3996.309938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3996.324411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3996.330434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3997.722761] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4000.017434] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4002.155894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4002.177194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4002.265202] Lustre: lustre-OST0000: new disk, initializing [ 4002.266909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4002.298266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.997428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4003.002059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4003.058174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4004.149605] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4008.256792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4008.289101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4008.348858] Lustre: lustre-OST0001: new disk, initializing [ 4008.350675] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4008.388993] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4010.207218] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4011.011105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4011.013845] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4011.041517] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4015.655233] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4018.316993] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4026.341351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4026.342407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4026.345045] Lustre: Skipped 3 previous similar messages [ 4026.346562] Lustre: Skipped 2 previous similar messages [ 4031.457292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4031.460241] Lustre: Skipped 3 previous similar messages [ 4032.390255] LustreError: 243265:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4032.419364] Lustre: server umount lustre-MDT0000 complete [ 4033.715603] LustreError: 239731:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038526 with bad export cookie 1493559083252005749 [ 4033.718944] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4033.719134] LustreError: 239731:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4033.790624] LustreError: 243467:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4033.792537] LustreError: 243467:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4033.850615] Lustre: server umount lustre-MDT0001 complete [ 4045.812115] LustreError: 243667:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4045.815750] LustreError: 243667:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4045.837689] Lustre: server umount lustre-OST0000 complete [ 4057.332902] LustreError: 243869:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4057.334725] LustreError: 243869:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4057.384243] Lustre: server umount lustre-OST0001 complete [ 4061.848895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4064.543528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4064.874802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4067.603460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4070.442451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4070.807943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4076.913407] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4080.694122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4080.701243] Lustre: lustre-MDT0000: reset Object Index mappings [ 4080.892196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4082.216097] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4085.140541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4085.146246] Lustre: lustre-MDT0001: reset Object Index mappings [ 4086.675858] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4087.661291] Lustre: 248284:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4089.998842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4090.137377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4090.139461] Lustre: Skipped 1 previous similar message [ 4091.176577] LustreError: 248640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4091.182076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4091.182253] LustreError: 248640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4092.098839] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4095.000365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4096.121600] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4097.130261] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4100.335368] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4101.636223] Lustre: 250130:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4110.817070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4110.819105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4110.822909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4111.847691] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4111.851156] Lustre: Skipped 2 previous similar messages [ 4113.992441] LustreError: 250237:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4113.994923] LustreError: 250237:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4114.027151] Lustre: server umount lustre-MDT0000 complete [ 4115.331762] LustreError: 247901:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038608 with bad export cookie 1493559083252012469 [ 4115.336886] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4115.453453] Lustre: server umount lustre-MDT0001 complete [ 4126.132141] LustreError: 250639:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4126.133783] LustreError: 250639:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4126.151339] Lustre: server umount lustre-OST0000 complete [ 4137.632363] Lustre: server umount lustre-OST0001 complete [ 4144.712407] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:03:57 (1767038637) [ 4150.378575] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4153.622379] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4157.215429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4159.286311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4161.364866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4163.588314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4166.480894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4166.511378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4166.608899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4166.622169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4166.661253] Lustre: lustre-MDT0000: new disk, initializing [ 4166.686215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4166.687832] Lustre: Skipped 1 previous similar message [ 4166.693696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4168.067741] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4172.343785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4172.364020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4172.400107] Lustre: 254357:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4172.514629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4174.094623] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4177.307886] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4178.461560] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4180.924645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4180.957422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4181.075491] Lustre: lustre-OST0000: new disk, initializing [ 4181.077208] Lustre: Skipped 1 previous similar message [ 4181.079742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4181.081765] Lustre: Skipped 2 previous similar messages [ 4183.033675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4183.036657] Lustre: Skipped 1 previous similar message [ 4183.038322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4183.075092] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4183.279567] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4186.369012] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4187.853524] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4187.920725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4189.344707] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4189.471336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4193.251284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4193.253479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4193.254650] LustreError: Skipped 1 previous similar message [ 4193.256204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4193.260193] Lustre: Skipped 1 previous similar message [ 4193.262158] Lustre: Skipped 4 previous similar messages [ 4196.271119] LustreError: 257296:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4196.275500] LustreError: 257296:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4196.310710] Lustre: server umount lustre-OST0000 complete [ 4198.369662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4198.370453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4198.371100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4198.371104] Lustre: Skipped 1 previous similar message [ 4198.377921] Lustre: Skipped 1 previous similar message [ 4203.676112] Lustre: server umount lustre-MDT0000 complete [ 4204.941425] LustreError: 254349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038697 with bad export cookie 1493559083252014562 [ 4204.945314] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4204.946533] LustreError: 254349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4205.086333] Lustre: server umount lustre-MDT0001 complete [ 4211.298933] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4214.769641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4214.916901] LustreError: 258871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4214.933901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4214.935573] Lustre: Skipped 2 previous similar messages [ 4216.286925] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4219.256924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4220.797062] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4221.766763] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4224.133364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4226.073103] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4228.939822] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4228.963899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4229.015914] Lustre: lustre-OST0001: new disk, initializing [ 4229.017969] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4230.719977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4230.723546] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4230.756056] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4230.956933] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4235.326050] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4236.492633] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4243.284323] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4243.286973] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4245.988043] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4245.991709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4245.993031] Lustre: Skipped 2 previous similar messages [ 4245.997263] Lustre: Skipped 6 previous similar messages [ 4251.204064] LustreError: 262377:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4251.206096] LustreError: 262377:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4251.236065] Lustre: server umount lustre-MDT0000 complete [ 4252.444210] LustreError: 258851:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038745 with bad export cookie 1493559083252015444 [ 4252.447925] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4252.450549] LustreError: 258851:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4252.568961] Lustre: server umount lustre-MDT0001 complete [ 4263.432323] Lustre: server umount lustre-OST0000 complete [ 4274.914446] Lustre: server umount lustre-OST0001 complete [ 4280.181397] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4283.643430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.810085] LustreError: 264155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4283.827741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.829744] Lustre: Skipped 3 previous similar messages [ 4285.123090] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4287.928365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4289.425352] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4290.390247] Lustre: 265264:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4292.614393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4293.799182] LustreError: 265619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4293.809895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4294.570534] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4297.444020] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4298.542873] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4299.377178] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4302.628407] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4303.883719] Lustre: 267120:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4310.576478] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4310.580533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4313.671030] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:06:46 (1767038806) [ 4328.928992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4328.930970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.934146] Lustre: Skipped 1 previous similar message [ 4328.935966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4328.938694] Lustre: Skipped 4 previous similar messages [ 4331.265862] LustreError: 267827:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4331.267766] LustreError: 267827:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4331.298634] Lustre: server umount lustre-MDT0000 complete [ 4332.535382] LustreError: 264136:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038825 with bad export cookie 1493559083252021744 [ 4332.541173] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4334.561495] LustreError: 265640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4334.567768] LustreError: 265640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4338.747584] Lustre: server umount lustre-MDT0001 complete [ 4346.111665] Lustre: server umount lustre-OST0000 complete [ 4353.497753] Lustre: server umount lustre-OST0001 complete [ 4355.567661] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4359.010997] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4362.778027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4364.895787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4366.970356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4369.148615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4374.410244] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4377.949748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4377.974415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4378.063773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4378.077044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4378.114875] Lustre: lustre-MDT0000: new disk, initializing [ 4378.150249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4379.470830] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4383.808673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4383.838696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4383.872357] Lustre: 271310:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4383.976590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4385.280883] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4389.817664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4389.841219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4391.343606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4391.908667] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4396.104141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4396.134703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4397.864184] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4398.065179] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4402.395277] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4412.537231] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 15:08:25 (1767038905) [ 4413.412581] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4413.414627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4413.421119] Lustre: Skipped 7 previous similar messages [ 4413.423899] Lustre: Skipped 9 previous similar messages [ 4419.357729] Lustre: server umount lustre-MDT0000 complete [ 4420.577201] LustreError: 271300:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038913 with bad export cookie 1493559083252029605 [ 4420.580833] LustreError: 271300:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4420.581682] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4420.689946] Lustre: server umount lustre-MDT0001 complete [ 4431.877638] Lustre: server umount lustre-OST0000 complete [ 4443.355280] Lustre: server umount lustre-OST0001 complete [ 4449.058701] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4452.562503] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4455.977411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4458.029173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4460.230230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4462.408235] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4465.209298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4465.232225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4465.323668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4465.326639] Lustre: Skipped 2 previous similar messages [ 4465.337364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4465.340676] Lustre: Skipped 4 previous similar messages [ 4465.380139] Lustre: lustre-MDT0000: new disk, initializing [ 4465.381520] Lustre: Skipped 3 previous similar messages [ 4465.409938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4465.412042] Lustre: Skipped 7 previous similar messages [ 4465.417909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4465.420509] Lustre: Skipped 3 previous similar messages [ 4466.811365] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4471.161338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4471.198882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4471.231830] Lustre: 278367:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4471.354789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4471.358295] Lustre: Skipped 2 previous similar messages [ 4472.703733] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4475.885625] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4477.108965] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4479.490075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4479.522310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4480.880802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4481.657408] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4484.793203] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4485.510784] Lustre: Mounted lustre-client [ 4485.829754] LustreError: 280873:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9065855ba800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4485.834922] LustreError: 280873:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4485.836890] LustreError: 280873:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 4485.857688] Lustre: Unmounted lustre-client [ 4490.726029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4490.730039] Lustre: Skipped 1 previous similar message [ 4490.732131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4490.736194] Lustre: Skipped 7 previous similar messages [ 4490.980667] Lustre: server umount lustre-OST0000 complete [ 4498.525995] Lustre: server umount lustre-MDT0000 complete [ 4499.768350] LustreError: 280114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767038992 with bad export cookie 1493559083252031131 [ 4499.772389] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4499.775523] LustreError: 280114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4499.890208] Lustre: server umount lustre-MDT0001 complete [ 4503.989522] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4505.103181] Key type lgssc unregistered [ 4547.232349] LNet: 282150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4547.235471] LNetError: 282150:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4617.894500] LNet: Removed LNI 192.168.204.113@tcp [ 4618.220124] Key type .llcrypt unregistered [ 4618.221563] Key type ._llcrypt unregistered [ 4624.263064] Key type ._llcrypt registered [ 4624.264651] Key type .llcrypt registered [ 4624.307736] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4630.962321] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4631.384968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4631.484019] alg: No test for adler32 (adler32-zlib) [ 4632.363284] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 4632.478046] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4634.064128] Key type lgssc registered [ 4634.497820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4637.516358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4639.672605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4641.714432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4643.897678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4644.511780] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:12:17 (1767039137) [ 4646.724728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4646.749586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4646.754486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.848160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4647.860944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4647.899815] Lustre: lustre-MDT0000: new disk, initializing [ 4647.922926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.929294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4649.147440] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4653.283526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4653.315024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4653.348389] Lustre: 285322:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4653.375220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4653.377554] Lustre: Skipped 1 previous similar message [ 4653.420793] Lustre: lustre-MDT0001: new disk, initializing [ 4653.461176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4653.473254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4653.476177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4654.833173] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4657.962975] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4659.234824] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4661.524444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4661.557814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4661.688441] Lustre: lustre-OST0000: new disk, initializing [ 4661.691197] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4661.731860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4663.743679] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4664.704839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4664.707934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4664.748082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4666.864944] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4675.042273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4675.042841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4675.044274] LustreError: Skipped 1 previous similar message [ 4675.048857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4675.051016] Lustre: Skipped 1 previous similar message [ 4676.845073] LustreError: 287763:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4676.880554] Lustre: server umount lustre-OST0000 complete [ 4679.137020] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4679.139027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4679.142811] Lustre: Skipped 1 previous similar message [ 4679.144558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4680.161423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4680.161613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.354953] LustreError: 287965:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4684.358200] LustreError: 287965:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4684.389467] Lustre: server umount lustre-MDT0000 complete [ 4685.603305] LustreError: 287066:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767039178 with bad export cookie 9012133751315154330 [ 4685.606096] LustreError: 287066:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4685.607223] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4685.675809] LustreError: 288167:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4685.677663] LustreError: 288167:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4685.732158] Lustre: server umount lustre-MDT0001 complete [ 4689.521307] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4690.628503] Key type lgssc unregistered [ 4690.770421] LNet: 288698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4690.772563] LNetError: 288698:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4690.780447] LNet: Removed LNI 192.168.204.113@tcp [ 4691.116191] Key type .llcrypt unregistered [ 4691.117819] Key type ._llcrypt unregistered [ 4725.433448] Key type ._llcrypt registered [ 4725.434492] Key type .llcrypt registered [ 4725.468756] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4725.563491] Key type .llcrypt unregistered [ 4725.564596] Key type ._llcrypt unregistered [ 4752.915373] Key type ._llcrypt registered [ 4752.916220] Key type .llcrypt registered [ 4752.958478] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4753.055416] Key type .llcrypt unregistered [ 4753.056474] Key type ._llcrypt unregistered [ 4764.191490] Key type ._llcrypt registered [ 4764.193024] Key type .llcrypt registered [ 4764.237957] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4764.349421] Key type .llcrypt unregistered [ 4764.350609] Key type ._llcrypt unregistered [ 4775.182834] Key type ._llcrypt registered [ 4775.183744] Key type .llcrypt registered [ 4775.217550] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4775.301412] Key type .llcrypt unregistered [ 4775.302506] Key type ._llcrypt unregistered [ 4802.557270] Key type ._llcrypt registered [ 4802.558582] Key type .llcrypt registered [ 4802.598924] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4802.707219] Key type .llcrypt unregistered [ 4802.708249] Key type ._llcrypt unregistered [ 4813.967551] Key type ._llcrypt registered [ 4813.968534] Key type .llcrypt registered [ 4813.999395] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4814.078430] Key type .llcrypt unregistered [ 4814.079430] Key type ._llcrypt unregistered [ 4825.490913] Key type ._llcrypt registered [ 4825.491832] Key type .llcrypt registered [ 4825.534810] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4825.644299] Key type .llcrypt unregistered [ 4825.645211] Key type ._llcrypt unregistered [ 4837.258800] Key type ._llcrypt registered [ 4837.260362] Key type .llcrypt registered [ 4837.303671] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4837.407574] Key type .llcrypt unregistered [ 4837.409081] Key type ._llcrypt unregistered [ 4849.198132] Key type ._llcrypt registered [ 4849.199187] Key type .llcrypt registered [ 4849.251762] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4849.378504] Key type .llcrypt unregistered [ 4849.380086] Key type ._llcrypt unregistered [ 4860.905245] Key type ._llcrypt registered [ 4860.906563] Key type .llcrypt registered [ 4860.949243] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4861.054353] Key type .llcrypt unregistered [ 4861.055295] Key type ._llcrypt unregistered [ 4884.670055] Key type ._llcrypt registered [ 4884.671606] Key type .llcrypt registered [ 4884.709551] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4884.811320] Key type .llcrypt unregistered [ 4884.812329] Key type ._llcrypt unregistered [ 4896.995739] Key type ._llcrypt registered [ 4896.997680] Key type .llcrypt registered [ 4897.042979] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4897.139282] Key type .llcrypt unregistered [ 4897.140333] Key type ._llcrypt unregistered [ 4908.729382] Key type ._llcrypt registered [ 4908.730401] Key type .llcrypt registered [ 4908.766345] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4908.863204] Key type .llcrypt unregistered [ 4908.864195] Key type ._llcrypt unregistered [ 4930.132316] Key type ._llcrypt registered [ 4930.134015] Key type .llcrypt registered [ 4930.169665] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4936.724472] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4937.270149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4937.290859] alg: No test for adler32 (adler32-zlib) [ 4938.152596] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 4938.252577] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4939.832143] Key type lgssc registered [ 4940.212984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4943.456994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4945.840342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4948.096406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4950.562146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4951.207602] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4779 sec ========== 15:17:24 (1767039444) [ 4951.874308] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:17:24 (1767039444) === [ 4953.267487] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:17:25 (1767039445) === [ 4967.085529] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4968.184832] Key type lgssc unregistered [ 4968.311336] LNet: 310392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4968.313281] LNetError: 310392:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4968.320418] LNet: Removed LNI 192.168.204.113@tcp [ 4968.619314] Key type .llcrypt unregistered [ 4968.620336] Key type ._llcrypt unregistered