[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478354341 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002377] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007440] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.010063] LSM: Security Framework initializing [ 0.011061] Yama: becoming mindful. [ 0.012000] SELinux: Initializing. [ 0.012072] *** VALIDATE selinux *** [ 0.020530] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025742] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027074] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029127] *** VALIDATE tmpfs *** [ 0.030475] *** VALIDATE proc *** [ 0.031251] *** VALIDATE cgroup *** [ 0.032013] *** VALIDATE cgroup2 *** [ 0.033287] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036000] Spectre V2 : User space: Vulnerable [ 0.036000] Speculative Store Bypass: Vulnerable [ 0.038220] debug: unmapping init [mem 0xffffffffa1459000-0xffffffffa1460fff] [ 0.040189] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041666] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042027] ... version: 2 [ 0.043013] ... bit width: 48 [ 0.044010] ... generic registers: 4 [ 0.045010] ... value mask: 0000ffffffffffff [ 0.046016] ... max period: 00007fffffffffff [ 0.047014] ... fixed-purpose events: 3 [ 0.048009] ... event mask: 000000070000000f [ 0.049319] rcu: Hierarchical SRCU implementation. [ 0.051621] smp: Bringing up secondary CPUs ... [ 0.052529] x86: Booting SMP configuration: [ 0.053026] .... node #0, CPUs: #1 #2 #3 [ 0.056099] smp: Brought up 1 node, 4 CPUs [ 0.058013] smpboot: Max logical packages: 1 [ 0.059018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.256033] node 0 deferred pages initialised in 194ms [ 0.259506] devtmpfs: initialized [ 0.260281] x86/mm: Memory block size: 128MB [ 0.262497] gcov: version magic: 0x41383552 [ 0.263660] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.267092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.270385] pinctrl core: initialized pinctrl subsystem [ 0.272198] [ 0.272824] ************************************************************* [ 0.275014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.277009] ** ** [ 0.279015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.281016] ** ** [ 0.284014] ** This means that this kernel is built to expose internal ** [ 0.287015] ** IOMMU data structures, which may compromise security on ** [ 0.289015] ** your system. ** [ 0.291017] ** ** [ 0.293016] ** If you see this message and you are not debugging the ** [ 0.296015] ** kernel, report this immediately to your vendor! ** [ 0.298013] ** ** [ 0.301014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.303011] ************************************************************* [ 0.305734] NET: Registered protocol family 16 [ 0.307425] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.310073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.313076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.316122] cpuidle: using governor menu [ 0.317675] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.320423] PCI: Using configuration type 1 for base access [ 0.321098] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.330100] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.333026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.336170] cryptd: max_cpu_qlen set to 1000 [ 0.339224] ACPI: Added _OSI(Module Device) [ 0.341027] ACPI: Added _OSI(Processor Device) [ 0.342018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.344023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.349275] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.357274] ACPI: Interpreter enabled [ 0.358084] ACPI: PM: (supports S0 S3 S4 S5) [ 0.360024] ACPI: Using IOAPIC for interrupt routing [ 0.362148] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.365505] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.375735] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.378074] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.381036] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.385160] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.390561] acpiphp: Slot [2] registered [ 0.392252] acpiphp: Slot [5] registered [ 0.393207] acpiphp: Slot [6] registered [ 0.395172] acpiphp: Slot [7] registered [ 0.396185] acpiphp: Slot [8] registered [ 0.398166] acpiphp: Slot [9] registered [ 0.399171] acpiphp: Slot [10] registered [ 0.401222] acpiphp: Slot [3] registered [ 0.402121] acpiphp: Slot [4] registered [ 0.404162] acpiphp: Slot [11] registered [ 0.406195] acpiphp: Slot [12] registered [ 0.407168] acpiphp: Slot [13] registered [ 0.409160] acpiphp: Slot [14] registered [ 0.411193] acpiphp: Slot [15] registered [ 0.413165] acpiphp: Slot [16] registered [ 0.414145] acpiphp: Slot [17] registered [ 0.416135] acpiphp: Slot [18] registered [ 0.417172] acpiphp: Slot [19] registered [ 0.419105] acpiphp: Slot [20] registered [ 0.421149] acpiphp: Slot [21] registered [ 0.422098] acpiphp: Slot [22] registered [ 0.424118] acpiphp: Slot [23] registered [ 0.425160] acpiphp: Slot [24] registered [ 0.427121] acpiphp: Slot [25] registered [ 0.428099] acpiphp: Slot [26] registered [ 0.429139] acpiphp: Slot [27] registered [ 0.431102] acpiphp: Slot [28] registered [ 0.432109] acpiphp: Slot [29] registered [ 0.434102] acpiphp: Slot [30] registered [ 0.435130] acpiphp: Slot [31] registered [ 0.437087] PCI host bridge to bus 0000:00 [ 0.438020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.440029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.443028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.445026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.448043] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.450039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.452219] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.455000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.458465] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.471021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.476953] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.480042] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.483027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.485033] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.488533] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.490795] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.493061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.496828] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.502019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.515019] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.521021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.528336] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.536025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.543019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.560017] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.570776] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.577026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.582020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.596023] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.605274] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.610013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.616016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.628016] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.639952] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.646017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.651015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.663022] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.672718] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.677024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.683018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.697018] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.707569] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.714020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.720022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.735018] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.745805] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.748450] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.750387] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.752390] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.754297] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.759160] iommu: Default domain type: Passthrough [ 0.761600] SCSI subsystem initialized [ 0.763214] ACPI: bus type USB registered [ 0.765188] usbcore: registered new interface driver usbfs [ 0.767080] usbcore: registered new interface driver hub [ 0.768183] usbcore: registered new device driver usb [ 0.770230] pps_core: LinuxPPS API ver. 1 registered [ 0.772022] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.775102] PTP clock support registered [ 0.778061] EDAC MC: Ver: 3.0.0 [ 0.780120] PCI: Using ACPI for IRQ routing [ 0.782028] NetLabel: Initializing [ 0.782969] NetLabel: domain hash size = 128 [ 0.783013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.784185] NetLabel: unlabeled traffic allowed by default [ 0.786179] vgaarb: loaded [ 0.788346] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.790025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.795866] clocksource: Switched to clocksource kvm-clock [ 0.906847] VFS: Disk quotas dquot_6.6.0 [ 0.908418] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.910771] *** VALIDATE ramfs *** [ 0.911984] *** VALIDATE hugetlbfs *** [ 0.913236] pnp: PnP ACPI init [ 0.915694] pnp: PnP ACPI: found 6 devices [ 0.938448] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.941723] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.944110] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.946393] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.948959] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.951066] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.953561] NET: Registered protocol family 2 [ 0.956049] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.960771] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.965946] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.971822] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.977780] TCP: Hash tables configured (established 65536 bind 65536) [ 0.981821] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.986114] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.988936] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.992457] NET: Registered protocol family 1 [ 0.994977] RPC: Registered named UNIX socket transport module. [ 0.997095] RPC: Registered udp transport module. [ 0.998837] RPC: Registered tcp transport module. [ 1.000285] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.002406] NET: Registered protocol family 44 [ 1.004142] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.006579] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.009171] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.011650] PCI: CLS 0 bytes, default 64 [ 1.013409] Unpacking initramfs... [ 2.422884] debug: unmapping init [mem 0xffff9a68bcc54000-0xffff9a68bffbffff] [ 2.426894] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.428859] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.431443] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.915920] Initialise system trusted keyrings [ 2.917954] Key type blacklist registered [ 2.920146] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.929139] zbud: loaded [ 2.932270] *** VALIDATE nfs *** [ 2.933635] *** VALIDATE nfs4 *** [ 2.940916] pstore: using deflate compression [ 2.944512] Platform Keyring initialized [ 3.052487] NET: Registered protocol family 38 [ 3.054199] Key type asymmetric registered [ 3.055863] Asymmetric key parser 'x509' registered [ 3.057519] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.060537] io scheduler mq-deadline registered [ 3.062223] io scheduler kyber registered [ 3.064068] io scheduler bfq registered [ 3.066118] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.069279] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.072214] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.074307] ACPI: Power Button [PWRF] [ 3.166314] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.258388] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.440937] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.534385] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.715436] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.744471] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.778816] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.783704] Non-volatile memory driver v1.3 [ 3.785703] Linux agpgart interface v0.103 [ 3.822438] virtio_blk virtio1: [vda] 132696 512-byte logical blocks (67.9 MB/64.8 MiB) [ 3.825197] vda: detected capacity change from 0 to 67940352 [ 3.841920] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.844991] vdb: detected capacity change from 0 to 1073741824 [ 3.861187] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.863988] vdc: detected capacity change from 0 to 2621440000 [ 3.878954] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.881825] vdd: detected capacity change from 0 to 2621440000 [ 3.896001] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.899058] vde: detected capacity change from 0 to 4294967296 [ 3.916461] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.919478] vdf: detected capacity change from 0 to 4294967296 [ 3.926601] libphy: Fixed MDIO Bus: probed [ 3.932216] usbcore: registered new interface driver usbserial_generic [ 3.934353] usbserial: USB Serial support registered for generic [ 3.936269] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.940743] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.942342] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.944496] mousedev: PS/2 mouse device common for all mice [ 3.947996] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.952065] rtc_cmos 00:05: RTC can wake from S4 [ 3.955492] rtc_cmos 00:05: registered as rtc0 [ 3.958933] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.959373] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.961578] intel_pstate: CPU model not supported [ 3.967880] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.971532] hid: raw HID events driver (C) Jiri Kosina [ 3.974426] usbcore: registered new interface driver usbhid [ 3.976238] usbhid: USB HID core driver [ 3.977656] drop_monitor: Initializing network drop monitor service [ 3.979702] Initializing XFRM netlink socket [ 3.981219] NET: Registered protocol family 10 [ 3.995031] Segment Routing with IPv6 [ 3.996657] NET: Registered protocol family 17 [ 4.000166] mpls_gso: MPLS GSO support [ 4.007559] RAS: Correctable Errors collector initialized. [ 4.010244] AVX version of gcm_enc/dec engaged. [ 4.012117] AES CTR mode by8 optimization enabled [ 4.088389] sched_clock: Marking stable (4088363103, 0)->(5019876022, -931512919) [ 4.092279] registered taskstats version 1 [ 4.094197] Loading compiled-in X.509 certificates [ 4.096272] zswap: loaded using pool lzo/zbud [ 4.119949] Key type big_key registered [ 4.132373] Key type encrypted registered [ 4.134337] ima: No TPM chip found, activating TPM-bypass! [ 4.136459] ima: Allocated hash algorithm: sha1 [ 4.138144] ima: No architecture policies found [ 4.140324] evm: Initialising EVM extended attributes: [ 4.142451] evm: security.selinux [ 4.143957] evm: security.ima [ 4.145623] evm: security.capability [ 4.147088] evm: HMAC attrs: 0x1 [ 4.149524] rtc_cmos 00:05: setting system clock to 2025-07-28 22:39:41 UTC (1753742381) [ 4.156774] debug: unmapping init [mem 0xffffffffa2403000-0xffffffffa25fffff] [ 4.159932] debug: unmapping init [mem 0xffffffffa1182000-0xffffffffa1458fff] [ 4.174085] Write protecting the kernel read-only data: 28672k [ 4.177514] debug: unmapping init [mem 0xffffffff9f803000-0xffffffff9f9fffff] [ 4.179823] debug: unmapping init [mem 0xffffffffa0114000-0xffffffffa01fffff] [ 4.219836] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.228954] systemd[1]: Detected virtualization kvm. [ 4.231028] systemd[1]: Detected architecture x86-64. [ 4.233137] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.260663] systemd[1]: No hostname configured. [ 4.262410] systemd[1]: Set hostname to . [ 4.264128] random: systemd: uninitialized urandom read (16 bytes read) [ 4.266094] systemd[1]: Initializing machine ID from random generator. [ 4.310509] random: ln: uninitialized urandom read (6 bytes read) [ 4.414412] random: systemd: uninitialized urandom read (16 bytes read) [ 4.416870] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.424182] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.429250] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.035571] device-mapper: uevent: version 1.0.3 [ 5.037812] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.708831] random: fast init done [ 5.743634] virtio_net virtio0 ens2: renamed from eth0 [ 5.856284] scsi host0: ata_piix [ 5.914461] scsi host1: ata_piix [ 5.916107] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.918599] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.106232] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.651631] random: crng init done [ 10.652899] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.048571] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.123862] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.351811] SELinux: Disabled at runtime. [ 12.411471] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.419498] systemd[1]: Detected virtualization kvm. [ 12.421178] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.850091] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.853189] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.860712] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.866160] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.869533] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.876768] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.885642] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ 13.008859] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.324215] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.599029] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.605227] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.735355] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.751263] EDAC sbridge: Ver: 1.1.2 [ 15.335576] Key type dns_resolver registered [ 15.637127] NFS: Registering the id_resolver key type [ 15.639243] Key type id_resolver registered [ 15.640835] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg107-server login: [ 41.480694] libcfs: loading out-of-tree module taints kernel. [ 41.491981] Key type ._llcrypt registered [ 41.493613] Key type .llcrypt registered [ 41.535700] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 57.296419] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 58.815311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 58.840296] alg: No test for adler32 (adler32-zlib) [ 60.149317] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 61.011674] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 61.015429] LNet: Accept secure, port 988 [ 62.776942] Key type lgssc registered [ 64.161083] hrtimer: interrupt took 2172205 ns [ 64.381574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.342265] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 84.424067] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.260805] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.550214] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.521946] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 121.943094] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 133.092270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.140680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 133.171629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 134.462328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 134.485984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 134.556680] Lustre: lustre-MDT0000: new disk, initializing [ 134.671759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 134.687807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 138.103680] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 149.088941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 149.138261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 149.187909] Lustre: 6472:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 149.225371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 149.228305] Lustre: Skipped 1 previous similar message [ 149.329266] Lustre: lustre-MDT0001: new disk, initializing [ 149.404562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.451428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 149.463197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 152.951155] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.814502] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 168.924751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 169.024977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 169.250200] Lustre: lustre-OST0000: new disk, initializing [ 169.277364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 169.400483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 175.289608] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.279092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 179.292652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 179.456276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 189.284612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 189.381065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 189.509713] Lustre: lustre-OST0001: new disk, initializing [ 189.519673] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 189.613329] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 195.123281] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.712745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 196.737945] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 196.795217] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 206.253415] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 215.533113] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 227.498306] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing check_logdir /tmp/testlogs/ [ 231.906701] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing yml_node [ 236.740748] Lustre: DEBUG MARKER: Client: 2.16.56.105 [ 239.384265] Lustre: DEBUG MARKER: MDS: 2.16.56.105 [ 242.478737] Lustre: DEBUG MARKER: OSS: 2.16.56.105 [ 244.250026] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jul 28 18:43:40 EDT 2025 [ 263.569985] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 264.882059] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 282.594315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 282.614788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 282.640741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.129591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.133146] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.493387] LustreError: 12140:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 285.604222] Lustre: server umount lustre-MDT0000 complete [ 288.789190] LustreError: 6464:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753742666 with bad export cookie 10295716930264038519 [ 288.794846] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 288.799866] LustreError: 6464:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 289.249592] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.268993] Lustre: Skipped 2 previous similar messages [ 289.276712] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 289.290815] Lustre: Skipped 4 previous similar messages [ 293.282238] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 295.005171] LustreError: 12342:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 295.007949] LustreError: 12342:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 295.242948] Lustre: server umount lustre-MDT0001 complete [ 299.337181] LustreError: 12544:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 299.348581] LustreError: 12544:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 299.421946] Lustre: server umount lustre-OST0000 complete [ 303.163090] LustreError: 12744:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 303.173145] LustreError: 12744:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 303.326309] Lustre: server umount lustre-OST0001 complete [ 309.559414] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 317.313238] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 326.929653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 333.662339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 339.214341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 345.527448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 351.807594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 351.908506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 352.239399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 352.271560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 352.335806] Lustre: lustre-MDT0000: new disk, initializing [ 352.438100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 352.467345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 355.811280] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 365.225652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 365.289118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 365.368596] Lustre: 14916:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 365.456419] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 365.465992] Lustre: Skipped 1 previous similar message [ 365.544860] Lustre: lustre-MDT0001: new disk, initializing [ 365.679618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 365.730113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 365.745042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 369.385928] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 376.972517] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 380.688371] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 388.081669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 388.136976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 388.399589] Lustre: lustre-OST0000: new disk, initializing [ 388.405786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 388.491585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.824326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 389.835928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 389.947396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 394.442880] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 401.666977] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 405.543255] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 405.739206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 409.913516] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 410.126347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 415.719068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 415.719244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 415.732610] LustreError: Skipped 1 previous similar message [ 415.739058] Lustre: Skipped 1 previous similar message [ 415.744699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 415.747390] Lustre: Skipped 2 previous similar messages [ 418.233296] LustreError: 17825:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 418.235980] LustreError: 17825:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 418.390325] Lustre: server umount lustre-OST0000 complete [ 421.861527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 421.869381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.954173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 425.957116] Lustre: Skipped 1 previous similar message [ 427.510832] LustreError: 18028:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 427.513901] LustreError: 18028:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 427.637498] Lustre: server umount lustre-MDT0000 complete [ 431.079829] LustreError: 15784:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 431.096850] LustreError: 15784:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 431.131832] LustreError: 15693:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753742808 with bad export cookie 10295716930264040052 [ 431.132724] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 431.147616] LustreError: 15693:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 431.563092] Lustre: server umount lustre-MDT0001 complete [ 436.663685] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:46:52 (1753742812) [ 453.244303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.660953] LustreError: 19364:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 453.761444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.295945] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 459.241388] LustreError: 19364:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.355359] LustreError: 19364:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.905212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 470.225961] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 476.233451] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 479.657116] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 481.743839] Lustre: Failing over lustre-MDT0000 [ 481.767561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 481.772887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.779324] Lustre: Skipped 1 previous similar message [ 481.787236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 481.937242] LustreError: 20694:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 481.946083] LustreError: 20694:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 482.035935] Lustre: server umount lustre-MDT0000 complete [ 483.819326] LustreError: 19364:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.835894] LustreError: 19364:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 496.111525] LustreError: 19363:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.127778] LustreError: 19363:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 500.195583] Lustre: 3639:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753742861/real 1753742861] req@ffff9a692f944e00 x1838932230823168/t0(0) o400->MGC192.168.201.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1753742877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 500.223583] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 500.700424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.434575] LustreError: 3637:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9a6802bbd180 x1838932230825600/t0(0) o250->MGC192.168.201.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 511.780895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 511.782032] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 511.839049] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 514.530337] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 522.206243] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 526.221822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.245422] Lustre: Skipped 1 previous similar message [ 526.260277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 526.271688] Lustre: Skipped 1 previous similar message [ 532.425426] LustreError: 21673:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 532.436317] LustreError: 21673:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 532.579314] Lustre: server umount lustre-MDT0000 complete [ 536.411633] LustreError: 19344:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753742913 with bad export cookie 10295716930264041466 [ 536.418519] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 536.423708] LustreError: 19344:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 536.760979] Lustre: server umount lustre-MDT0001 complete [ 544.571853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.061836] LustreError: 22352:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.081286] LustreError: 22352:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 545.148653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 545.152699] Lustre: Skipped 2 previous similar messages [ 548.839307] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 557.084953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 560.541569] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 565.817244] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 569.854857] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 572.491466] Lustre: Failing over lustre-MDT0000 [ 572.827643] Lustre: server umount lustre-MDT0000 complete [ 572.897978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 572.909960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 572.923866] Lustre: Skipped 1 previous similar message [ 580.067841] LustreError: 22352:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 580.107447] LustreError: 22352:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 591.330302] Lustre: 3638:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753742952/real 1753742952] req@ffff9a692f946680 x1838932230850560/t0(0) o400->MGC192.168.201.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1753742968 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 591.360322] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 592.525526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.573684] Lustre: Evicted from MGS (at 192.168.201.107@tcp) after server handle changed from 0x0 to 0x8ee1bbffc05d0b9d [ 601.597098] Lustre: MGC192.168.201.107@tcp: Connection restored to (at 0@lo) [ 601.600866] Lustre: Skipped 1 previous similar message [ 602.006915] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 602.915396] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 602.922984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 602.941838] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 605.726156] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 612.638756] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 617.443620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 617.447562] Lustre: Skipped 2 previous similar messages [ 622.413597] LustreError: 24657:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 622.417320] LustreError: 24657:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 622.550575] Lustre: server umount lustre-MDT0000 complete [ 626.799904] LustreError: 22332:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743004 with bad export cookie 10295716930264042397 [ 626.801427] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 626.822746] LustreError: 22332:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 627.332717] Lustre: server umount lustre-MDT0001 complete [ 636.910805] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:50:12 (1753743012) [ 654.710986] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 663.947646] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 674.164614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 680.406439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 686.724887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 693.487412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 702.385611] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 712.233911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 712.301352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.546140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 712.585801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 712.632828] Lustre: lustre-MDT0000: new disk, initializing [ 712.706457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 712.710235] Lustre: Skipped 2 previous similar messages [ 712.731899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 716.791608] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 727.613073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 727.661507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 727.724762] Lustre: 28305:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 728.118048] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 731.818733] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 740.298310] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 744.487221] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 752.525356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 752.598451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 754.546227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 754.592668] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 754.624328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 754.636547] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 759.786405] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 766.839597] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 772.097256] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 795.622601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 795.626295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 795.636235] Lustre: Skipped 3 previous similar messages [ 795.644050] LustreError: Skipped 1 previous similar message [ 795.649131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 795.658904] Lustre: Skipped 1 previous similar message [ 798.668171] LustreError: 30948:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 798.677641] LustreError: 30948:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 798.876234] Lustre: server umount lustre-OST0000 complete [ 808.677352] Lustre: server umount lustre-MDT0000 complete [ 810.977337] LustreError: 28312:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 810.997471] LustreError: 28312:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 13 previous similar messages [ 812.695926] LustreError: 28296:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743190 with bad export cookie 10295716930264042796 [ 812.701880] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 812.731539] LustreError: 28296:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 813.432689] Lustre: server umount lustre-MDT0001 complete [ 825.634685] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 828.642948] Key type lgssc unregistered [ 828.984226] LNet: 31878:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.055928] LNet: Removed LNI 192.168.201.107@tcp [ 831.306856] Key type .llcrypt unregistered [ 831.309217] Key type ._llcrypt unregistered [ 855.461574] Key type ._llcrypt registered [ 855.463399] Key type .llcrypt registered [ 855.602473] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 857.522033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 857.749147] alg: No test for adler32 (adler32-zlib) [ 858.969951] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 859.265229] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 859.274238] LNet: Accept secure, port 988 [ 860.960659] Key type lgssc registered [ 862.781082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 876.110625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 876.167557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 878.486332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.099312] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 887.778201] LustreError: 33261:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.785909] LustreError: 33261:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 892.421768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 892.785129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 896.959512] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 902.694618] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 906.123904] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 913.809699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 914.313339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.399164] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 922.073330] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 923.537295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 928.940168] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 944.606608] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:55:20 (1753743320) [ 946.432206] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 948.524950] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 956.788587] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:55:32 (1753743332) [ 965.567141] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:55:41 (1753743341) [ 966.683164] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 975.508657] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 975.512230] Lustre: Skipped 2 previous similar messages [ 977.367805] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:55:53 (1753743353) [ 981.033622] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 981.037608] Lustre: Skipped 3 previous similar messages [ 1000.862586] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1000.866283] Lustre: Skipped 6 previous similar messages [ 1003.919303] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:56:19 (1753743379) [ 1007.402607] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1011.096528] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1020.182598] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:56:36 (1753743396) [ 1022.380799] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1022.385839] Lustre: Skipped 3 previous similar messages [ 1038.126904] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:56:53 (1753743413) [ 1056.740824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1056.743107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.759155] Lustre: Skipped 1 previous similar message [ 1056.761522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.570277] LustreError: 39819:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1058.759415] Lustre: server umount lustre-OST0000 complete [ 1066.978512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1066.983057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1066.987512] Lustre: Skipped 1 previous similar message [ 1066.987814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1066.990592] Lustre: Skipped 2 previous similar messages [ 1066.992529] LustreError: Skipped 1 previous similar message [ 1068.885587] LustreError: 40020:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1068.890467] LustreError: 40020:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1068.948969] Lustre: server umount lustre-MDT0000 complete [ 1072.099577] LustreError: 33256:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.128930] LustreError: 33256:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1072.556633] LustreError: 37336:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743449 with bad export cookie 1110218909260248184 [ 1072.563313] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1072.705284] LustreError: 40222:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1072.724076] LustreError: 40222:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1072.999053] Lustre: server umount lustre-MDT0001 complete [ 1084.517991] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1087.503725] Key type lgssc unregistered [ 1087.840689] LNet: 40748:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.874255] LNet: Removed LNI 192.168.201.107@tcp [ 1089.733185] Key type .llcrypt unregistered [ 1089.735922] Key type ._llcrypt unregistered [ 1107.143582] Key type ._llcrypt registered [ 1107.145261] Key type .llcrypt registered [ 1107.274523] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 1119.406428] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1120.131958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1120.161840] alg: No test for adler32 (adler32-zlib) [ 1121.233110] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 1121.485184] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1121.493214] LNet: Accept secure, port 988 [ 1123.216184] Key type lgssc registered [ 1124.310605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1133.555389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1140.771030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1146.845817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1153.162095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1159.341911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1159.387222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1159.408204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.661333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1160.701062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1160.816863] Lustre: lustre-MDT0000: new disk, initializing [ 1160.942405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1160.984255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1164.657543] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1174.243395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1174.301821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1174.361746] Lustre: 43857:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1174.414788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1174.417234] Lustre: Skipped 1 previous similar message [ 1174.488345] Lustre: lustre-MDT0001: new disk, initializing [ 1174.603744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1174.643579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1174.654134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1178.005540] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1183.918784] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1186.536887] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1193.134949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1193.204972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1193.411022] Lustre: lustre-OST0000: new disk, initializing [ 1193.414171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1193.504797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.379865] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1203.756512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1203.761771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1203.849293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1204.934449] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1208.849664] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1209.095686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1212.549470] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1212.698556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1219.042086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1219.054018] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.075518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1219.553407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1219.556516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.596159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1220.612154] LustreError: 46765:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1220.807118] Lustre: server umount lustre-OST0000 complete [ 1224.676245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1224.692704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1224.699088] Lustre: Skipped 1 previous similar message [ 1229.800783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1229.811843] Lustre: Skipped 2 previous similar messages [ 1230.715481] LustreError: 46967:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1230.729111] LustreError: 46967:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1230.839973] Lustre: server umount lustre-MDT0000 complete [ 1234.338638] LustreError: 43849:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743611 with bad export cookie 16692513647368051221 [ 1234.347354] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1234.352826] LustreError: 43849:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1234.505060] LustreError: 47168:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1234.511549] LustreError: 47168:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1234.771334] Lustre: server umount lustre-MDT0001 complete [ 1244.018678] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 19:00:19 (1753743619) [ 1251.019963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.544658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.115588] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1256.937344] LustreError: 47887:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.944916] LustreError: 47887:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1261.027296] LustreError: 47888:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.737777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1264.140163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1267.399685] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1272.223707] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1275.298303] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1281.467705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1281.789156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.531484] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1292.479992] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1301.139052] LustreError: 50078:0:(llog.c:430:llog_init_handle()) llog type is not specified! [ 1301.879658] Lustre: Modifying parameter general.timeout=1 in log params [ 1303.398578] Lustre: Modifying parameter general.timeout=3 in log params [ 1303.405362] Lustre: Skipped 1 previous similar message [ 1305.927367] Lustre: Modifying parameter general.timeout=6 in log params [ 1305.929845] Lustre: Skipped 2 previous similar messages [ 1310.081180] Lustre: Modifying parameter general.timeout=11 in log params [ 1310.083728] Lustre: Skipped 4 previous similar messages [ 1318.431953] Lustre: Modifying parameter general.timeout=22 in log params [ 1318.436879] Lustre: Skipped 10 previous similar messages [ 1335.226839] Lustre: Modifying parameter general.timeout=42 in log params [ 1335.229033] Lustre: Skipped 19 previous similar messages [ 1367.795659] Lustre: Modifying parameter general.timeout=81 in log params [ 1367.799034] Lustre: Skipped 38 previous similar messages [ 1417.563946] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 19:03:13 (1753743793) [ 1420.117993] Lustre: 57015:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1421.694533] Lustre: 57113:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1424.774950] Lustre: 57323:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1424.790956] Lustre: 57323:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 1432.090955] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1432.097211] Lustre: Skipped 58 previous similar messages [ 1451.721745] Lustre: 58897:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1451.737502] Lustre: 58897:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 2 previous similar messages [ 1451.786606] LustreError: 58897:0:(mgs_handler.c:990:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1455.725202] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 19:03:52 (1753743832) [ 1466.850702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.859716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1466.870234] Lustre: Skipped 1 previous similar message [ 1466.875797] Lustre: Skipped 1 previous similar message [ 1472.906802] LustreError: 59598:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1473.024305] LustreError: 59598:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1473.284290] Lustre: server umount lustre-MDT0000 complete [ 1477.091644] LustreError: 47884:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1477.101932] LustreError: 47867:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743854 with bad export cookie 16692513647368052103 [ 1477.107861] LustreError: 47884:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1477.110738] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1477.150750] LustreError: 47867:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1477.282494] LustreError: 59799:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1477.285189] LustreError: 59799:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1477.506801] Lustre: server umount lustre-MDT0001 complete [ 1492.266152] LustreError: 60000:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1492.268857] LustreError: 60000:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1492.528547] Lustre: server umount lustre-OST0000 complete [ 1500.704594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1503.811241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1507.156947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1510.593683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1517.329487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1517.672033] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1517.994903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.179126] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1529.234557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1529.333478] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1529.368540] Lustre: 61452:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1533.090986] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1538.359278] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1542.188769] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1548.231197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1548.356460] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1553.460405] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1558.136523] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1559.832026] Lustre: 63665:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1561.569339] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1561.572122] Lustre: Skipped 15 previous similar messages [ 1569.762145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1569.762990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1569.770062] LustreError: Skipped 1 previous similar message [ 1569.782455] Lustre: Skipped 1 previous similar message [ 1569.792027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1569.799700] Lustre: Skipped 3 previous similar messages [ 1572.343621] LustreError: 63922:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1572.345951] LustreError: 63922:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1572.462316] Lustre: server umount lustre-OST0000 complete [ 1575.907102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1580.004087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1580.010360] Lustre: Skipped 1 previous similar message [ 1581.953262] Lustre: server umount lustre-MDT0000 complete [ 1585.127909] LustreError: 61460:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1585.156089] LustreError: 61460:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1585.575967] LustreError: 61443:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743962 with bad export cookie 16692513647368055099 [ 1585.578954] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1585.628833] LustreError: 61443:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1585.932207] Lustre: server umount lustre-MDT0001 complete [ 1596.311912] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1599.221543] Key type lgssc unregistered [ 1599.598875] LNet: 64854:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1600.682076] LNet: Removed LNI 192.168.201.107@tcp [ 1601.828505] Key type .llcrypt unregistered [ 1601.831737] Key type ._llcrypt unregistered [ 1625.772698] Key type ._llcrypt registered [ 1625.774277] Key type .llcrypt registered [ 1625.919902] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1627.094338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1627.176110] alg: No test for adler32 (adler32-zlib) [ 1628.386231] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 1628.801481] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1628.805582] LNet: Accept secure, port 988 [ 1630.520499] Key type lgssc registered [ 1631.861064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1640.951217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1640.968858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1642.606562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1646.884850] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1652.194179] LustreError: 66238:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.213228] LustreError: 66238:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1656.384454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1657.222661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1661.973476] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1668.542767] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1673.648778] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1682.323688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1682.844483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.333203] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1695.507620] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1706.456652] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1717.730930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1717.734492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1717.789138] Lustre: Skipped 1 previous similar message [ 1717.828701] Lustre: Skipped 2 previous similar messages [ 1721.023186] LustreError: 68928:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1721.176720] Lustre: server umount lustre-MDT0000 complete [ 1722.850698] LustreError: 66238:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.865165] LustreError: 66238:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1725.159642] LustreError: 66219:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744102 with bad export cookie 13952026388808338437 [ 1725.165247] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1725.181601] LustreError: 66219:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1725.352243] LustreError: 69132:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1725.355102] LustreError: 69132:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1725.765943] Lustre: server umount lustre-MDT0001 complete [ 1740.177527] LustreError: 69335:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1740.188159] LustreError: 69335:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1740.361486] Lustre: server umount lustre-OST0000 complete [ 1749.358613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1753.033186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1756.836446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1760.716560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1768.083259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1768.382940] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1768.427248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1768.692734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1772.593916] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1780.645933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1780.762761] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1780.799745] Lustre: 70787:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1781.222677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1784.402554] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1789.078158] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1792.517144] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1798.813649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1798.945945] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1799.345847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.442304] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1809.355829] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1810.786133] Lustre: 73012:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1812.366829] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1818.595643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1818.599253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.607767] Lustre: Skipped 1 previous similar message [ 1818.612044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1819.105631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1821.811185] LustreError: 73269:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1821.814902] LustreError: 73269:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1821.989229] Lustre: server umount lustre-OST0000 complete [ 1827.301941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1827.307442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1827.314354] Lustre: Skipped 1 previous similar message [ 1827.318439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1827.322320] Lustre: Skipped 2 previous similar messages [ 1829.346020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.347244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1829.371616] Lustre: Skipped 1 previous similar message [ 1831.339048] LustreError: 73471:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1831.342095] LustreError: 73471:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1831.416851] Lustre: server umount lustre-MDT0000 complete [ 1834.467138] LustreError: 70826:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1834.490478] LustreError: 70826:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1834.814778] LustreError: 70780:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744212 with bad export cookie 13952026388808339270 [ 1834.817373] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1834.831152] LustreError: 70780:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1835.217183] Lustre: server umount lustre-MDT0001 complete [ 1844.662950] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1847.360233] Key type lgssc unregistered [ 1847.646720] LNet: 74198:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1848.691576] LNet: Removed LNI 192.168.201.107@tcp [ 1849.410140] Key type .llcrypt unregistered [ 1849.411685] Key type ._llcrypt unregistered [ 1872.039313] Key type ._llcrypt registered [ 1872.041034] Key type .llcrypt registered [ 1872.158812] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1873.194687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1873.211019] alg: No test for adler32 (adler32-zlib) [ 1874.333775] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 1874.553291] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1874.557448] LNet: Accept secure, port 988 [ 1876.209120] Key type lgssc registered [ 1877.173297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1883.855430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1883.886113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.619546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.102056] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1894.893375] LustreError: 75578:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.909384] LustreError: 75578:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1896.191933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1896.658772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1899.828000] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1904.394660] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1907.632846] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1913.018465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1913.432609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1917.500783] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1921.375977] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1937.146677] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1937.859323] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1939.152911] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1939.156147] Lustre: Skipped 1 previous similar message [ 1941.543297] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1941.546208] Lustre: Skipped 3 previous similar messages [ 1946.059503] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1946.063296] Lustre: Skipped 7 previous similar messages [ 1954.478750] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1954.483356] Lustre: Skipped 13 previous similar messages [ 1970.788849] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1970.793391] Lustre: Skipped 26 previous similar messages [ 2003.874108] Lustre: 83334:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2004.531116] Lustre: Disabling parameter general.jobid_var= in log params [ 2004.533540] Lustre: Skipped 43 previous similar messages [ 2005.856991] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:13:02 (1753744382) [ 2007.097264] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2008.335381] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:13:04 (1753744384) [ 2012.136622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2012.138380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.144280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2012.156413] Lustre: Skipped 1 previous similar message [ 2017.015262] LustreError: 83629:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2017.121479] Lustre: server umount lustre-OST0000 complete [ 2022.370945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.372182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2022.379226] Lustre: Skipped 1 previous similar message [ 2022.382764] Lustre: Skipped 1 previous similar message [ 2025.693182] LustreError: 83830:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2025.700580] LustreError: 83830:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2025.762113] Lustre: server umount lustre-MDT0000 complete [ 2027.492754] LustreError: 75578:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.502600] LustreError: 75578:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2028.071070] LustreError: 77194:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744405 with bad export cookie 13196955419207903556 [ 2028.071525] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.077147] LustreError: 77194:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2028.162974] LustreError: 84031:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2028.166169] LustreError: 84031:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2028.305904] Lustre: server umount lustre-MDT0001 complete [ 2035.463950] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 2037.373604] Key type lgssc unregistered [ 2037.609133] LNet: 84557:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.632328] LNet: Removed LNI 192.168.201.107@tcp [ 2039.291611] Key type .llcrypt unregistered [ 2039.293242] Key type ._llcrypt unregistered [ 2042.331059] Key type ._llcrypt registered [ 2042.333433] Key type .llcrypt registered [ 2042.406514] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2045.235700] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules [ 2045.824556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2045.890033] alg: No test for adler32 (adler32-zlib) [ 2046.851224] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 2047.048032] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 2047.052968] LNet: Accept secure, port 988 [ 2048.672158] Key type lgssc registered [ 2049.543376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2050.404638] LustreError: 85695:0:(lu_object.c:1385:lu_context_key_register()) cfs_race id 60d sleeping [ 2064.226478] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2070.725511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2072.611530] LustreError: 85695:0:(lu_object.c:1385:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2072.649955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2074.162635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.011074] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2083.523900] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:14:20 (1753744460) [ 2084.324708] LustreError: 86705:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2084.336084] LustreError: 86705:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2086.767650] LustreError: 86736:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2086.773266] LustreError: 86736:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2086.779458] LustreError: 86736:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [ 2086.885957] LustreError: 87504:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2087.062135] Lustre: server umount lustre-MDT0000 complete [ 2103.044665] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 2106.572778] Key type lgssc unregistered [ 2106.976446] LNet: 88188:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.009468] LNet: Removed LNI 192.168.201.107@tcp [ 2108.813105] Key type .llcrypt unregistered [ 2108.814598] Key type ._llcrypt unregistered [ 2128.057221] Key type ._llcrypt registered [ 2128.061758] Key type .llcrypt registered [ 2128.244220] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2130.117797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2130.189285] alg: No test for adler32 (adler32-zlib) [ 2131.510705] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 2131.822183] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 2131.830237] LNet: Accept secure, port 988 [ 2133.528232] Key type lgssc registered [ 2134.831646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2145.302551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2145.342905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.100540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.468387] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2157.540095] LustreError: 89378:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.564096] LustreError: 89378:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2159.981914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2160.746961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2164.635745] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2169.538963] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2173.509313] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2179.974851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2180.547927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2186.485322] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2192.254779] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2235.434501] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:16:51 (1753744611) [ 2237.129891] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2238.923297] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:16:55 (1753744615) [ 2242.530630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2242.536158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.544043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2243.044140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.046745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2243.061347] Lustre: Skipped 1 previous similar message [ 2243.069819] Lustre: Skipped 1 previous similar message [ 2248.041895] LustreError: 92270:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2248.105192] Lustre: server umount lustre-MDT0000 complete [ 2248.166232] LustreError: 90790:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.174716] LustreError: 90790:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2251.867210] LustreError: 89358:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744629 with bad export cookie 10889054834958977509 [ 2251.879958] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2251.896749] LustreError: 89358:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2252.056027] LustreError: 92474:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2252.058972] LustreError: 92474:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2252.311511] Lustre: server umount lustre-MDT0001 complete [ 2266.514922] LustreError: 92675:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2266.517977] LustreError: 92675:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2266.839528] Lustre: server umount lustre-OST0000 complete [ 2276.559263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.937607] LustreError: 93304:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.130217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2280.933292] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2282.473065] LustreError: 93303:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.500816] LustreError: 93303:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2287.595974] LustreError: 93303:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.456916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2289.095618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2293.025110] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2298.651856] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2302.967967] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2309.585216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2316.971450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2317.067739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2317.254771] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2317.259260] LustreError: 93292:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2317.293703] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2317.308513] LustreError: 95154:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -98 [ 2317.325694] LustreError: 95154:0:(tgt_mount.c:1766:server_put_super()) no obd lustre-OST0000 [ 2317.328968] LustreError: 95154:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2317.529646] Lustre: server umount lustre-OST0000 complete [ 2317.534047] LustreError: 95154:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2326.938344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2327.208380] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2327.213079] LustreError: 93292:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2327.256091] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2327.271814] LustreError: 95620:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -98 [ 2327.274444] LustreError: 95620:0:(tgt_mount.c:1766:server_put_super()) no obd lustre-OST0000 [ 2327.277343] LustreError: 95620:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2327.565992] Lustre: server umount lustre-OST0000 complete [ 2327.568379] LustreError: 95620:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2330.811053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2337.780622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2338.047341] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2338.056357] Lustre: Found index 0 for lustre-OST0000, updating log [ 2338.067957] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2338.179974] Lustre: lustre-OST0000: new disk, initializing [ 2338.187606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2338.476859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2344.481760] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2348.093976] Lustre: Failing over lustre-OST0000 [ 2348.134210] LustreError: 96704:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2348.144245] LustreError: 96704:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2348.394569] Lustre: server umount lustre-OST0000 complete [ 2355.170264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2355.179946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.202710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2355.210251] Lustre: Skipped 1 previous similar message [ 2357.217135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2357.218356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.404657] LustreError: 96904:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2362.435395] LustreError: 96904:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2362.531031] Lustre: server umount lustre-MDT0000 complete [ 2364.391375] LustreError: 93304:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.425905] LustreError: 93304:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2365.976306] LustreError: 93286:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744743 with bad export cookie 10889054834958980260 [ 2365.985902] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2365.990373] LustreError: 93286:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2366.360253] Lustre: server umount lustre-MDT0001 complete [ 2375.824792] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:19:11 (1753744751) [ 2392.360753] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2403.739483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2404.224894] LustreError: 98466:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.384122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2408.402943] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2417.899981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2422.893938] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2425.974587] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2433.089245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2434.214977] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2440.999497] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2443.443289] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2443.446940] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2443.459625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2448.873157] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2448.898555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2449.894137] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2449.923463] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2449.929890] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.107@tcp (at 0@lo) [ 2449.949253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2450.870993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2450.970571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2451.030809] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2451.140180] Lustre: lustre-OST0001: new disk, initializing [ 2451.154453] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2451.513861] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2451.520172] Lustre: Skipped 2 previous similar messages [ 2457.918227] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2458.233370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2458.238991] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2458.332570] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2465.941570] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2472.430493] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2484.198807] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.203050] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2484.220120] Lustre: Skipped 4 previous similar messages [ 2484.221145] Lustre: Skipped 2 previous similar messages [ 2487.963534] LustreError: 101685:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2487.980529] LustreError: 101685:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2488.100085] Lustre: server umount lustre-MDT0001 complete [ 2489.315069] LustreError: 99938:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.331584] LustreError: 99938:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 2491.296769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2498.424294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2498.541896] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2498.557615] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2498.613270] Lustre: 98454:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2499.783058] LustreError: 102254:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2499.787653] LustreError: 102254:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2499.793634] LustreError: 98457:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2499.867884] LustreError: 102259:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.201.107@tcp: cfg command failed: rc = -17 [ 2499.875775] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2499.875775] [ 2503.303912] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2504.679448] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 2513.719588] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 19:21:29 (1753744889) [ 2532.028977] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2549.619028] Lustre: 104279:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2573.810351] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2577.222500] Lustre: 105413:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2668.010214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2668.022714] LustreError: Skipped 1 previous similar message [ 2668.027272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.041874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2668.049166] Lustre: Skipped 2 previous similar messages [ 2671.634813] LustreError: 105870:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2671.645731] LustreError: 105870:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2671.741625] Lustre: server umount lustre-MDT0000 complete [ 2673.642763] LustreError: 99938:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.651326] LustreError: 99938:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 8 previous similar messages [ 2675.277114] LustreError: 98446:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753745052 with bad export cookie 10889054834958981254 [ 2675.280443] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2675.292575] LustreError: 98446:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2675.626814] Lustre: server umount lustre-MDT0001 complete [ 2689.621144] Lustre: server umount lustre-OST0000 complete [ 2703.728715] LustreError: 106475:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2703.743316] LustreError: 106475:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 2704.156958] Lustre: server umount lustre-OST0001 complete [ 2710.139040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2718.519783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2719.518293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2728.380723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2737.709449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2738.970371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2764.075195] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2776.078259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.091403] Lustre: lustre-MDT0000: reset Object Index mappings [ 2776.645808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.649454] Lustre: Skipped 1 previous similar message [ 2780.844910] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2789.517294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2789.538518] Lustre: lustre-MDT0001: reset Object Index mappings [ 2794.242311] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2797.009479] Lustre: 110462:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2803.629246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2809.555621] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2815.652840] LustreError: 110820:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2815.678037] LustreError: 110820:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2815.684156] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2818.211901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2820.058561] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2821.110524] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2821.120220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 2824.627539] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2833.072224] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2836.181228] Lustre: 112304:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2886.336859] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:27:42 (1753745262) [ 2912.738658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2912.745485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2912.759169] Lustre: Skipped 3 previous similar messages [ 2912.767204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2912.772950] Lustre: Skipped 3 previous similar messages [ 2918.690337] LustreError: 112653:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2918.699739] LustreError: 112653:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2918.819734] Lustre: server umount lustre-MDT0000 complete [ 2922.472994] LustreError: 111634:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753745299 with bad export cookie 10889054834959028350 [ 2922.473659] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2922.485341] LustreError: 111634:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2929.090370] Lustre: server umount lustre-MDT0001 complete [ 2939.120670] Lustre: server umount lustre-OST0000 complete [ 2949.360145] Lustre: server umount lustre-OST0001 complete [ 2955.095516] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 2962.971419] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2973.887837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2981.980594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2989.158933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2996.115918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3002.641736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3002.714345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.919470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3002.956969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3003.036302] Lustre: lustre-MDT0000: new disk, initializing [ 3003.120520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.129808] Lustre: Skipped 3 previous similar messages [ 3003.151214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3006.687136] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3013.821901] Lustre: server umount lustre-MDT0000 complete [ 3016.737844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3023.771926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3023.890667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.343360] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3024.375104] Lustre: 116513:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3024.401779] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3029.517292] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3035.761756] Lustre: server umount lustre-MDT0000 complete [ 3046.933466] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:30:22 (1753745422) [ 3049.098415] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3051.452073] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:30:27 (1753745427) [ 3069.450924] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3080.655252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.362706] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3093.970490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3094.027276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3094.096476] Lustre: 118517:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3094.150126] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3094.153111] Lustre: Skipped 1 previous similar message [ 3094.289630] Lustre: lustre-MDT0001: new disk, initializing [ 3094.491569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3094.506418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3099.629331] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3104.641622] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3110.284672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3110.364218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3110.590815] Lustre: lustre-OST0000: new disk, initializing [ 3110.595602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3111.871481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3111.880749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3111.979121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3115.353860] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3123.452182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3123.512098] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3123.679128] Lustre: lustre-OST0001: new disk, initializing [ 3123.682628] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3124.994943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3125.006097] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3125.112125] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3128.149775] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3135.667737] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3138.528262] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3804.716244] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:43:00 (1753746180) [ 3821.543421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3821.544652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3821.548847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3821.548857] Lustre: Skipped 12 previous similar messages [ 3821.561850] Lustre: Skipped 7 previous similar messages [ 3826.664394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3826.666430] Lustre: Skipped 3 previous similar messages [ 3833.312141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3833.430890] LustreError: 127533:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3833.439478] LustreError: 127533:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 3833.558784] Lustre: server umount lustre-MDT0000 complete [ 3836.540578] LustreError: 118509:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746213 with bad export cookie 10889054834959094654 [ 3836.543212] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3836.552744] LustreError: 118509:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3836.898372] LustreError: 118522:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3836.898844] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.900090] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3836.900097] Lustre: Skipped 4 previous similar messages [ 3836.912285] LustreError: 118522:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 3836.933480] Lustre: Skipped 2 previous similar messages [ 3843.062399] Lustre: server umount lustre-MDT0001 complete [ 3852.672455] LustreError: 127939:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3852.675738] LustreError: 127939:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3852.723930] Lustre: server umount lustre-OST0000 complete [ 3870.176183] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3870.430597] Lustre: server umount lustre-OST0001 complete [ 3875.695545] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 3882.011328] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3888.143202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3892.419775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3896.939869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3902.034848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3907.442720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3907.500805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3907.694549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3907.720444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3907.780360] Lustre: lustre-MDT0000: new disk, initializing [ 3907.863072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3907.867028] Lustre: Skipped 5 previous similar messages [ 3907.895172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3910.934547] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3919.745681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3919.798820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3919.858946] Lustre: 130319:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3919.906683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3919.909185] Lustre: Skipped 1 previous similar message [ 3919.982133] Lustre: lustre-MDT0001: new disk, initializing [ 3920.106959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3920.114567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3923.085762] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3929.517592] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3932.151194] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3936.754560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3936.813901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3937.017581] Lustre: lustre-OST0000: new disk, initializing [ 3937.023610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3938.290481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3938.298552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3938.377061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3941.106694] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3946.676053] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3954.492627] Lustre: lustre-MDD0000: changelog on [ 3956.574914] Lustre: lustre-MDD0001: changelog on [ 5331.858477] Lustre: lustre-MDD0001: changelog off [ 5332.762456] Lustre: 130325:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5333.157552] Lustre: lustre-MDD0000: changelog off [ 5336.037419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5336.037539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5336.039174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5336.039180] Lustre: Skipped 3 previous similar messages [ 5336.042385] Lustre: Skipped 1 previous similar message [ 5341.154783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5341.158303] Lustre: Skipped 1 previous similar message [ 5341.302073] LustreError: 140046:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5341.303889] LustreError: 140046:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5341.418778] Lustre: server umount lustre-OST0000 complete [ 5343.714298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5343.718540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5346.273333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5346.273941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5346.276129] Lustre: Skipped 1 previous similar message [ 5349.464648] LustreError: 140248:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5349.466746] LustreError: 140248:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5349.549734] Lustre: server umount lustre-MDT0000 complete [ 5351.284482] LustreError: 130311:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747728 with bad export cookie 10889054834959912884 [ 5351.287679] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5351.289615] LustreError: 130311:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5351.477120] Lustre: server umount lustre-MDT0001 complete [ 5357.277582] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 5358.551399] Key type lgssc unregistered [ 5358.705834] LNet: 140976:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5359.718528] LNet: Removed LNI 192.168.201.107@tcp [ 5360.089176] Key type .llcrypt unregistered [ 5360.090644] Key type ._llcrypt unregistered [ 5368.596337] Key type ._llcrypt registered [ 5368.597531] Key type .llcrypt registered [ 5368.647868] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 5375.723174] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5376.162111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5376.182541] alg: No test for adler32 (adler32-zlib) [ 5377.064456] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 5377.171535] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 5377.174813] LNet: Accept secure, port 988 [ 5378.776201] Key type lgssc registered [ 5379.239150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5383.448169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5386.377740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5388.942965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5391.775131] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5394.700811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5394.723461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5394.733142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5395.825583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5395.841616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5395.881249] Lustre: lustre-MDT0000: new disk, initializing [ 5395.910271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5395.918541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5397.410278] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5402.157044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5402.190039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5402.232793] Lustre: 144107:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5402.264571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5402.267051] Lustre: Skipped 1 previous similar message [ 5402.308348] Lustre: lustre-MDT0001: new disk, initializing [ 5402.350761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5402.369760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5402.374699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5404.000082] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5407.841424] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5409.357325] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5412.151053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5412.191734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5412.304145] Lustre: lustre-OST0000: new disk, initializing [ 5412.306718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5412.342642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5414.864194] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5418.675651] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5419.519398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5419.524365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5419.571748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5426.663720] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_module obdecho/obdecho [ 5427.427548] LustreError: 146704:0:(echo_client.c:670:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5427.431659] LustreError: 146704:0:(obd_config.c:779:class_setup()) setup ec failed (-22) [ 5428.148683] LustreError: 146800:0:(echo_client.c:1878:echo_md_handler()) Can not resolve the path : rc = -2 [ 5433.313249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5433.317074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.323084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5434.850288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5434.850663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5434.857372] Lustre: Skipped 1 previous similar message [ 5434.860741] Lustre: Skipped 1 previous similar message [ 5438.722576] LustreError: 147140:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5438.763290] Lustre: server umount lustre-MDT0000 complete [ 5440.368588] LustreError: 145850:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747817 with bad export cookie 4447601114053499044 [ 5440.373165] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5440.378591] LustreError: 145850:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5440.448369] LustreError: 147343:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5440.456731] LustreError: 147343:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5440.570950] Lustre: server umount lustre-MDT0001 complete [ 5452.594735] LustreError: 147543:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5452.596697] LustreError: 147543:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5452.680457] Lustre: server umount lustre-OST0000 complete [ 5461.494565] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 5465.565164] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5469.660882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5472.466519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5475.198404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5478.012150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5480.967336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5481.002894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.112268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5481.135414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5481.177279] Lustre: lustre-MDT0000: new disk, initializing [ 5481.222473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5481.236819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5482.901737] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5488.147806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5488.189603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5488.232519] Lustre: 150470:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5488.258977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5488.261292] Lustre: Skipped 1 previous similar message [ 5488.301458] Lustre: lustre-MDT0001: new disk, initializing [ 5488.342194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5488.356936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5488.361164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5489.941439] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5493.560449] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5495.076703] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5497.908474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5497.938322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5498.047483] Lustre: lustre-OST0000: new disk, initializing [ 5498.050198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5498.089369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5500.452873] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5502.975744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5502.980057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5503.022053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5503.957331] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5505.572580] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5505.664137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5507.131614] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5507.221255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5508.067387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5508.070251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.071811] LustreError: Skipped 1 previous similar message [ 5508.073601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5508.077389] Lustre: Skipped 1 previous similar message [ 5508.079596] Lustre: Skipped 1 previous similar message [ 5509.998104] LustreError: 153374:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5510.001223] LustreError: 153374:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5510.069607] Lustre: server umount lustre-OST0000 complete [ 5513.186264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5513.187238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5513.192302] Lustre: Skipped 1 previous similar message [ 5513.195665] Lustre: Skipped 1 previous similar message [ 5517.760381] LustreError: 153575:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5517.762499] LustreError: 153575:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5517.800988] Lustre: server umount lustre-MDT0000 complete [ 5518.305285] LustreError: 151250:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5518.311815] LustreError: 151250:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 5519.304290] LustreError: 152203:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747896 with bad export cookie 4447601114053500010 [ 5519.307729] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5519.310429] LustreError: 152203:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5519.458446] Lustre: server umount lustre-MDT0001 complete [ 5520.910504] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:11:37 (1753747897) [ 5521.547205] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5522.268299] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:11:39 (1753747899) [ 5525.270417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.451444] LustreError: 154448:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.482336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5526.993830] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5530.516036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5530.593068] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5530.595849] Lustre: Skipped 1 previous similar message [ 5532.199965] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5534.419840] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5535.923867] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5538.720884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5541.245721] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5543.407872] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5556.810393] LustreError: 156696:0:(ldlm_resource.c:981:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd0:0x1:0x0].0x3ade9a31 (00000000e0ac531d) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5559.023725] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.7@tcp (stopping) [ 5560.802694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5560.807835] Lustre: Skipped 2 previous similar messages [ 5562.960042] LustreError: 156696:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5562.965414] LustreError: 156696:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5564.134118] LustreError: 154442:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5565.050418] Lustre: server umount lustre-MDT0000 complete [ 5566.591330] LustreError: 154427:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747943 with bad export cookie 4447601114053500892 [ 5566.593656] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5566.595776] LustreError: 154427:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5569.253650] LustreError: 156186:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5569.262913] LustreError: 156186:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5571.042510] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5572.834992] Lustre: server umount lustre-MDT0001 complete [ 5575.652502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5580.955496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5581.080066] LustreError: 157958:0:(llog.c:1602:llog_backup()) MGC192.168.201.107@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5581.085457] Lustre: 157958:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.201.107@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5589.733212] LustreError: 157966:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5589.742562] LustreError: 157966:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5599.971555] LustreError: 157966:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5599.978504] LustreError: 157966:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5616.097127] LustreError: 157966:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5616.103753] LustreError: 157966:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 14 previous similar messages [ 5642.208305] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5642.210505] Lustre: 157958:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.201.107@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5650.148274] LustreError: 157966:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5650.155160] LustreError: 157966:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 20 previous similar messages [ 5671.904471] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5671.907202] Lustre: 154439:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.201.107@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5682.662528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5682.666410] Lustre: Skipped 2 previous similar messages [ 5684.296312] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5686.756592] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 5687.545245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5692.384211] LustreError: 154439:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 5712.352888] LustreError: 142325:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000892a254f x1838937805003264/t0(0) o250->MGC192.168.201.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5714.098238] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5716.266308] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5716.894703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5717.478216] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 5718.198712] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5721.946112] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5732.834343] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5732.835226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5732.840744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5732.844797] Lustre: Skipped 7 previous similar messages [ 5732.905416] LustreError: 159430:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5732.907767] LustreError: 159430:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5732.939230] Lustre: server umount lustre-MDT0000 complete [ 5734.346368] LustreError: 155161:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748111 with bad export cookie 4447601114053507388 [ 5734.349026] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5734.350833] LustreError: 155161:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5734.500995] Lustre: server umount lustre-MDT0001 complete [ 5746.314585] Lustre: server umount lustre-OST0000 complete [ 5754.554026] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 5758.614077] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5762.852671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5765.638748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5767.963141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5770.550300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5773.141048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5773.167068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5773.266791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5773.280957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5773.318917] Lustre: lustre-MDT0000: new disk, initializing [ 5773.350949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5773.353346] Lustre: Skipped 1 previous similar message [ 5773.360187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5774.849031] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5779.746101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5779.778107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5779.822548] Lustre: 162760:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5779.948202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5781.583660] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5785.330574] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5786.845831] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5789.770189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5789.809958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5792.374937] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5793.279303] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5793.327348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5796.049568] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5797.698228] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5797.788433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5799.387371] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5799.486277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5803.492862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5803.494574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5803.495378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5803.499133] Lustre: Skipped 3 previous similar messages [ 5803.506967] LustreError: Skipped 1 previous similar message [ 5806.316105] LustreError: 165666:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5806.317909] LustreError: 165666:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5806.385651] Lustre: server umount lustre-OST0000 complete [ 5814.049422] Lustre: server umount lustre-MDT0000 complete [ 5815.653553] LustreError: 162752:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748193 with bad export cookie 4447601114053524181 [ 5815.656449] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5815.658695] LustreError: 162752:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5815.837620] Lustre: server umount lustre-MDT0001 complete [ 5819.341747] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:16:36 (1753748196) [ 5821.981775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5822.139136] LustreError: 166739:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5822.146144] LustreError: 166739:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 45 previous similar messages [ 5823.630594] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5827.207217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5828.957780] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5831.202661] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5832.717401] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5835.462041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5837.920955] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5840.206088] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5855.285835] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:17:12 (1753748232) [ 5857.253876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5862.518820] Lustre: server umount lustre-OST0000 complete [ 5870.167815] Lustre: server umount lustre-MDT0000 complete [ 5871.575493] LustreError: 166718:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748248 with bad export cookie 4447601114053525063 [ 5871.578656] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5871.580124] LustreError: 166718:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5871.646617] LustreError: 169587:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5871.648937] LustreError: 169587:0:(obd_class.h:479:obd_check_dev()) Skipped 27 previous similar messages [ 5871.735185] Lustre: server umount lustre-MDT0001 complete [ 5876.872713] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 5878.069370] Key type lgssc unregistered [ 5878.213530] LNet: 170113:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5879.270337] LNet: Removed LNI 192.168.201.107@tcp [ 5879.614305] Key type .llcrypt unregistered [ 5879.616040] Key type ._llcrypt unregistered [ 5888.237872] Key type ._llcrypt registered [ 5888.239094] Key type .llcrypt registered [ 5888.277944] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5888.814536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5888.825362] alg: No test for adler32 (adler32-zlib) [ 5889.703111] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 5889.804861] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 5889.807727] LNet: Accept secure, port 988 [ 5891.408182] Key type lgssc registered [ 5891.808467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5894.690226] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5894.698521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5912.160291] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5918.304210] LustreError: 171301:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.107@tcp: failed processing log, type 4: rc = -110 [ 5924.448444] LustreError: 171301:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5924.453141] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5924.456125] LustreError: MGC192.168.201.107@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5924.460044] LustreError: 171301:0:(tgt_mount.c:1527:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5924.463756] LustreError: 171301:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -22 [ 5924.467205] LustreError: 171301:0:(tgt_mount.c:1766:server_put_super()) no obd lustre-OST0000 [ 5924.470459] LustreError: 171301:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 5924.524044] Lustre: server umount lustre-OST0000 complete [ 5924.525952] LustreError: 171301:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5928.062021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5929.274294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5930.790232] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5934.157081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5934.313487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5935.861039] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5938.128510] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5939.655719] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5942.392538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5942.487958] LustreError: 173383:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5942.493284] Lustre: 173383:0:(mgc_request_server.c:610:mgc_llog_local_copy()) MGC192.168.201.107@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5942.555316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5944.787904] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5947.159625] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5949.923522] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5949.926526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5949.930669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5949.936015] Lustre: Skipped 2 previous similar messages [ 5954.175372] LustreError: 174028:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5954.177349] LustreError: 174028:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5954.208503] Lustre: server umount lustre-MDT0000 complete [ 5955.042163] LustreError: 171803:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5955.048305] LustreError: 171803:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 5955.588320] LustreError: 171783:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748332 with bad export cookie 10654666653107636111 [ 5955.591292] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5955.592564] LustreError: 171783:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5955.658243] LustreError: 174232:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5955.660600] LustreError: 174232:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5955.742654] Lustre: server umount lustre-MDT0001 complete [ 5967.409189] LustreError: 174432:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5967.411624] LustreError: 174432:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5967.502249] Lustre: server umount lustre-OST0000 complete [ 5971.096153] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 5975.447249] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5979.385803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5982.052818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5984.530218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5986.964632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5989.578491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5989.603934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5989.698968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5989.717707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5989.759310] Lustre: lustre-MDT0000: new disk, initializing [ 5989.800493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5989.809541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5991.248190] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5996.009877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5996.034613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5996.077415] Lustre: 176750:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5996.104464] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5996.107013] Lustre: Skipped 1 previous similar message [ 5996.144966] Lustre: lustre-MDT0001: new disk, initializing [ 5996.181587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5996.195437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5996.199702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5997.776881] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6001.446912] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6002.979931] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6005.693726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6005.721456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6005.819856] Lustre: lustre-OST0000: new disk, initializing [ 6005.823018] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6005.854716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6006.358391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6006.362757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6006.399567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6008.047753] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6011.511783] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6013.164614] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6013.253585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6014.770918] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6014.857475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6016.993638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6016.995571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6016.997532] LustreError: Skipped 1 previous similar message [ 6017.002746] Lustre: Skipped 2 previous similar messages [ 6017.003569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6017.009123] Lustre: Skipped 1 previous similar message [ 6021.741620] LustreError: 179657:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6021.744108] LustreError: 179657:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6021.809476] Lustre: server umount lustre-OST0000 complete [ 6027.233325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6027.233731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6027.234198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6027.237507] Lustre: Skipped 2 previous similar messages [ 6029.345232] Lustre: server umount lustre-MDT0000 complete [ 6030.840146] LustreError: 178484:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748408 with bad export cookie 10654666653107636797 [ 6030.842956] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6030.845184] LustreError: 178484:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6030.913924] LustreError: 180060:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6030.915594] LustreError: 180060:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6031.001213] Lustre: server umount lustre-MDT0001 complete [ 6034.583889] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:20:11 (1753748411) [ 6041.257031] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6042.435521] Key type lgssc unregistered [ 6042.571677] LNet: 181237:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6043.622542] LNet: Removed LNI 192.168.201.107@tcp [ 6043.965399] Key type .llcrypt unregistered [ 6043.966579] Key type ._llcrypt unregistered [ 6052.495933] Key type ._llcrypt registered [ 6052.497534] Key type .llcrypt registered [ 6052.546320] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6053.001798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6053.012279] alg: No test for adler32 (adler32-zlib) [ 6053.880332] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 6053.979918] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6053.982764] LNet: Accept secure, port 988 [ 6055.576135] Key type lgssc registered [ 6055.990118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6059.093243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6059.102079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6060.281757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6061.801796] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6065.166563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6065.313533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6066.861200] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6068.999936] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6070.354867] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6073.190719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6073.264477] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6073.266135] LustreError: 184027:0:(llog.c:1642:llog_backup()) MGC192.168.201.107@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6073.269745] Lustre: 184027:0:(mgc_request_server.c:607:mgc_llog_local_copy()) MGC192.168.201.107@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6073.313312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6075.466579] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6077.606883] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6080.481761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6080.483917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6080.485470] Lustre: Skipped 2 previous similar messages [ 6080.488130] Lustre: Skipped 2 previous similar messages [ 6084.541383] LustreError: 184672:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6084.569850] Lustre: server umount lustre-MDT0000 complete [ 6085.601563] LustreError: 182396:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6085.607064] LustreError: 182396:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 6085.900184] LustreError: 182381:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748463 with bad export cookie 6274507345819569322 [ 6085.903277] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6085.907153] LustreError: 182381:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6085.961508] LustreError: 184876:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6085.963930] LustreError: 184876:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6086.041161] Lustre: server umount lustre-MDT0001 complete [ 6096.818473] LustreError: 185077:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6096.821374] LustreError: 185077:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6096.885078] Lustre: server umount lustre-OST0000 complete [ 6100.556495] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 6104.961721] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6108.679202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6111.478853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6114.374136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6117.130185] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6119.921944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6119.952775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6120.043097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6120.058136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6120.095081] Lustre: lustre-MDT0000: new disk, initializing [ 6120.125517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6120.134920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6121.703308] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6126.571986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6126.594261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6126.630284] Lustre: 187395:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6126.650176] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6126.651370] Lustre: Skipped 1 previous similar message [ 6126.653015] LustreError: 188155:0:(llog.c:1642:llog_backup()) MGC192.168.201.107@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6126.656154] LustreError: 188155:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 6126.658144] Lustre: 188155:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.201.107@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6126.666421] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6126.668967] Lustre: Skipped 1 previous similar message [ 6126.704223] Lustre: lustre-MDT0001: new disk, initializing [ 6126.742816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6126.756843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6126.761250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6128.240901] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6131.694134] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6133.178391] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6135.756953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6135.784784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6135.892636] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6135.894585] Lustre: Skipped 1 previous similar message [ 6135.896248] LustreError: 189104:0:(llog.c:1642:llog_backup()) MGC192.168.201.107@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6135.899881] LustreError: 189104:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 6135.902477] Lustre: 189104:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.201.107@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6135.906858] Lustre: 189104:0:(mgc_request_server.c:630:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6135.917144] Lustre: lustre-OST0000: new disk, initializing [ 6135.919451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6135.950865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6138.235303] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6141.870736] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6142.971382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6142.975592] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6143.015085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6143.572536] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6143.664709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6145.200716] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6145.287419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6148.066398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6148.071957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.077884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6148.080784] Lustre: Skipped 1 previous similar message [ 6152.109839] LustreError: 190301:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6152.112177] LustreError: 190301:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6152.183566] Lustre: server umount lustre-OST0000 complete [ 6157.792978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6157.795606] LustreError: Skipped 1 previous similar message [ 6157.797447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6157.802249] Lustre: Skipped 1 previous similar message [ 6157.804097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6159.807496] LustreError: 190502:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6159.810420] LustreError: 190502:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6159.845912] Lustre: server umount lustre-MDT0000 complete [ 6161.279202] LustreError: 187387:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748538 with bad export cookie 6274507345819570022 [ 6161.282294] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6161.283847] LustreError: 187387:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6161.445412] Lustre: server umount lustre-MDT0001 complete [ 6164.748958] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:22:21 (1753748541) [ 6171.181349] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6175.035404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6175.180690] LustreError: 192062:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6175.195984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6176.638483] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6179.910387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6181.356077] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6182.376599] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6184.858111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6187.368255] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6190.802758] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6190.833338] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6190.903284] Lustre: lustre-OST0001: new disk, initializing [ 6190.905845] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6193.265467] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6198.799608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6198.803755] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6198.845920] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6199.369879] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6202.362407] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6208.154137] Lustre: DEBUG MARKER: ADD OST3 [ 6210.111253] loop: module loaded [ 6210.132142] loop0: detected capacity change from 0 to 204800000 [ 6210.276341] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6210.281246] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6210.286317] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6210.346701] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6212.863167] loop0: detected capacity change from 0 to 204800000 [ 6215.153759] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6215.179962] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6215.251148] Lustre: lustre-OST0003: new disk, initializing [ 6215.254466] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6215.304863] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6215.308033] Lustre: Skipped 3 previous similar messages [ 6215.532792] Lustre: DEBUG MARKER: STOP OST3 [ 6216.339829] Lustre: Failing over lustre-OST0003 [ 6216.358084] LustreError: 196288:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 6216.360973] LustreError: 196288:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6216.377595] Lustre: server umount lustre-OST0003 complete [ 6216.712814] LustreError: 193502:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6217.745928] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6221.564765] Lustre: DEBUG MARKER: START OST3 again [ 6221.800407] LustreError: 193504:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6221.807148] LustreError: 193504:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6224.199119] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6226.167810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6226.171990] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6226.184489] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 6226.524329] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6241.075216] Lustre: Failing over lustre-OST0003 [ 6241.099527] LustreError: 198314:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 6241.102641] LustreError: 198314:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6241.123979] Lustre: server umount lustre-OST0003 complete [ 6241.761956] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6241.762478] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6241.764868] LustreError: Skipped 1 previous similar message [ 6241.769745] Lustre: Skipped 1 previous similar message [ 6241.772762] LustreError: 193505:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6241.778533] LustreError: 193505:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6249.964152] LustreError: 194802:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6249.970834] LustreError: 194802:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6255.077980] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6255.080388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6255.083300] Lustre: Skipped 2 previous similar messages [ 6255.087359] Lustre: Skipped 5 previous similar messages [ 6260.193582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6260.198681] Lustre: Skipped 3 previous similar messages [ 6268.384190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6268.476402] Lustre: server umount lustre-MDT0000 complete [ 6270.028550] LustreError: 194308:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748647 with bad export cookie 6274507345819570883 [ 6270.031896] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6270.033796] LustreError: 194308:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6270.433723] LustreError: 193522:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.434392] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6270.435447] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6270.435454] Lustre: Skipped 4 previous similar messages [ 6270.440296] LustreError: 193522:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 6270.445604] Lustre: Skipped 2 previous similar messages [ 6276.235901] LustreError: 198786:0:(obd_class.h:479:obd_check_dev()) Device 29 not setup [ 6276.238179] LustreError: 198786:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6276.352885] Lustre: server umount lustre-MDT0001 complete [ 6277.889102] Lustre: server umount lustre-OST0000 complete [ 6279.527510] Lustre: server umount lustre-OST0001 complete [ 6282.073856] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 6285.230856] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6289.928159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6292.722696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6295.250075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6297.838212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6300.673782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6300.698266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6300.806093] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6300.821481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6300.866245] Lustre: lustre-MDT0000: new disk, initializing [ 6300.895426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6300.898417] Lustre: Skipped 1 previous similar message [ 6300.906720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6302.500459] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6307.393233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6307.423761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6307.464978] Lustre: 201353:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6307.607313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6309.213914] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6312.672754] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6314.048520] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6316.775123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6316.806109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6316.918695] Lustre: lustre-OST0000: new disk, initializing [ 6316.920877] Lustre: Skipped 1 previous similar message [ 6316.923936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6316.927359] Lustre: Skipped 2 previous similar messages [ 6318.651609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6318.655488] Lustre: Skipped 1 previous similar message [ 6318.658098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6318.694471] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6318.954536] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6322.316360] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6323.782089] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6323.857405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6325.246623] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6325.324531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6328.802522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6328.803111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6328.806781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6328.809783] Lustre: Skipped 1 previous similar message [ 6328.811459] Lustre: Skipped 3 previous similar messages [ 6332.136467] Lustre: server umount lustre-OST0000 complete [ 6333.408901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6333.411812] LustreError: Skipped 1 previous similar message [ 6339.546326] Lustre: server umount lustre-MDT0000 complete [ 6340.944394] LustreError: 202128:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748718 with bad export cookie 6274507345819573151 [ 6340.947577] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6340.948795] LustreError: 202128:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6341.014360] LustreError: 204665:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6341.017501] LustreError: 204665:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 6341.099411] Lustre: server umount lustre-MDT0001 complete [ 6342.465352] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 20:25:19 (1753748719) [ 6348.060286] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 6351.631887] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6355.183852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6357.640305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6359.852880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6362.235292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6364.846343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6364.867961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6364.953785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6364.965914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6365.001144] Lustre: lustre-MDT0000: new disk, initializing [ 6365.027703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6366.516152] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6370.886516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6370.908322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6370.941738] Lustre: 207487:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6371.050249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6372.412463] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6375.646245] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6376.898551] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6379.298523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6379.325463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6381.095895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6381.515527] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6384.743922] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6386.238852] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6386.324960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6387.820184] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6387.903552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6391.266136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6391.267886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6391.269177] LustreError: Skipped 1 previous similar message [ 6391.274060] Lustre: Skipped 2 previous similar messages [ 6391.274771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6391.279500] Lustre: Skipped 5 previous similar messages [ 6394.726792] Lustre: server umount lustre-OST0000 complete [ 6402.142147] Lustre: server umount lustre-MDT0000 complete [ 6403.591664] LustreError: 207478:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748780 with bad export cookie 6274507345819574033 [ 6403.594844] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6403.596692] LustreError: 207478:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6403.771162] Lustre: server umount lustre-MDT0001 complete [ 6406.806087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6406.993749] LustreError: 211271:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6407.001164] LustreError: 211271:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6408.534843] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6411.920081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6413.536802] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6415.527508] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6416.930147] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6419.465231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6421.829723] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6424.096381] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6442.469738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6442.472646] LustreError: Skipped 1 previous similar message [ 6447.288788] Lustre: server umount lustre-OST0000 complete [ 6454.806212] Lustre: server umount lustre-MDT0000 complete [ 6456.132881] LustreError: 213819:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748833 with bad export cookie 6274507345819574915 [ 6456.136669] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6456.137480] LustreError: 213819:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6456.301000] Lustre: server umount lustre-MDT0001 complete [ 6558.690135] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6559.786456] Key type lgssc unregistered [ 6559.922334] LNet: 214582:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6560.934261] LNet: Removed LNI 192.168.201.107@tcp [ 6561.224707] Key type .llcrypt unregistered [ 6561.225581] Key type ._llcrypt unregistered [ 6569.532127] Key type ._llcrypt registered [ 6569.533286] Key type .llcrypt registered [ 6569.568975] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 6576.086747] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6576.424347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6576.477901] alg: No test for adler32 (adler32-zlib) [ 6577.360747] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 6577.450922] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6577.452622] LNet: Accept secure, port 988 [ 6579.032131] Key type lgssc registered [ 6579.374399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6582.512046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6584.990324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6587.133694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6589.591437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6592.154201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6592.172294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6592.178420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6593.263762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6593.278298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6593.315359] Lustre: lustre-MDT0000: new disk, initializing [ 6593.337808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6593.345655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6594.740202] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6599.125276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6599.149038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6599.184898] Lustre: 217866:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6599.213775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6599.215963] Lustre: Skipped 1 previous similar message [ 6599.257781] Lustre: lustre-MDT0001: new disk, initializing [ 6599.295499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6599.309303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6599.312742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6600.695740] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6604.024144] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6605.404668] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6607.925854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6607.952534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6608.054917] Lustre: lustre-OST0000: new disk, initializing [ 6608.058588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6608.091919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6609.466309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6609.470677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6609.507205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6610.073159] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6613.567419] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6615.237667] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6615.357540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6616.864779] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6616.942646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6619.618347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6619.619587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6619.621044] LustreError: Skipped 1 previous similar message [ 6619.621899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6619.626564] Lustre: Skipped 1 previous similar message [ 6619.631563] Lustre: Skipped 1 previous similar message [ 6619.642819] LustreError: 220770:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6619.721853] Lustre: server umount lustre-OST0000 complete [ 6624.740125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6624.741444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6624.745816] Lustre: Skipped 1 previous similar message [ 6624.749197] Lustre: Skipped 1 previous similar message [ 6627.203577] LustreError: 220972:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6627.205553] LustreError: 220972:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6627.233720] Lustre: server umount lustre-MDT0000 complete [ 6628.627258] LustreError: 217856:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749005 with bad export cookie 2297757568880126729 [ 6628.630403] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6628.632310] LustreError: 217856:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6628.695404] LustreError: 221174:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6628.698775] LustreError: 221174:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6628.799836] Lustre: server umount lustre-MDT0001 complete [ 6635.902458] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6639.721351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6639.902482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6641.348971] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6644.608739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6644.745304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6646.081204] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6647.081625] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6649.456128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6651.579753] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6654.837480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6654.859599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6654.917144] Lustre: lustre-OST0001: new disk, initializing [ 6654.919601] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6654.949982] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6654.952240] Lustre: Skipped 1 previous similar message [ 6656.396574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6656.400117] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6656.440160] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6657.096836] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6663.230738] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6666.275780] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6676.239601] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 20:30:53 (1753749053) [ 6679.105571] LustreError: 225687:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6679.108490] LustreError: 225687:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6679.140583] Lustre: server umount lustre-MDT0000 complete [ 6680.455223] LustreError: 222317:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749057 with bad export cookie 2297757568880127597 [ 6680.458910] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6680.459204] LustreError: 222317:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6680.605516] Lustre: server umount lustre-MDT0001 complete [ 6698.208152] Lustre: 216086:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753749059/real 1753749059] req@ffff9a680c6d8000 x1838939063273088/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1753749075 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6698.218047] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6701.216153] Lustre: 216088:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753749062/real 1753749062] req@ffff9a680c6db480 x1838939063273472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1753749078 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6701.225798] Lustre: 216088:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6702.579690] LustreError: 226087:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6702.581616] LustreError: 226087:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6702.598751] Lustre: server umount lustre-OST0000 complete [ 6703.971257] Lustre: server umount lustre-OST0001 complete [ 6706.202544] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 6709.629996] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6713.367108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6715.847690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6718.153233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6720.564659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6723.018819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6723.045800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6723.145405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6723.163421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6723.201711] Lustre: lustre-MDT0000: new disk, initializing [ 6723.232362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6723.240172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6724.608409] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6729.025731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6729.054892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6729.094816] Lustre: 228456:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6729.205648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6730.586647] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6733.940377] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6735.429275] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6738.061852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6738.090751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6738.187085] Lustre: lustre-OST0000: new disk, initializing [ 6738.188387] Lustre: Skipped 1 previous similar message [ 6738.190224] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6738.192568] Lustre: Skipped 2 previous similar messages [ 6739.258289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6739.261591] Lustre: Skipped 1 previous similar message [ 6739.263981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6739.296191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6740.173186] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6743.456667] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6744.805261] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6744.874580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6746.275115] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6746.350059] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6749.037293] LustreError: 231365:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6749.039966] LustreError: 231365:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6749.116341] Lustre: server umount lustre-OST0000 complete [ 6754.785980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6754.786171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6754.786691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6754.791208] Lustre: Skipped 2 previous similar messages [ 6759.905787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6759.908240] Lustre: Skipped 2 previous similar messages [ 6759.935902] Lustre: server umount lustre-MDT0000 complete [ 6761.212705] LustreError: 228446:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749138 with bad export cookie 2297757568880129697 [ 6761.216633] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6761.224502] LustreError: 228446:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6770.659997] LustreError: 231767:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6770.661586] LustreError: 231767:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6770.749618] Lustre: server umount lustre-MDT0001 complete [ 6773.618206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6773.759846] LustreError: 232244:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6773.765522] LustreError: 232244:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6773.782765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6773.785830] Lustre: Skipped 2 previous similar messages [ 6775.119312] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6778.056186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6779.475300] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6781.294542] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6782.613604] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6785.113392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6787.370409] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6789.402850] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6936.035310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6936.036076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6936.040396] Lustre: Skipped 1 previous similar message [ 6939.626521] LustreError: 234512:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6939.628171] LustreError: 234512:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6939.691504] Lustre: server umount lustre-OST0000 complete [ 6946.273020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6946.275564] Lustre: Skipped 4 previous similar messages [ 6947.096142] Lustre: server umount lustre-MDT0000 complete [ 6948.343588] LustreError: 232224:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749325 with bad export cookie 2297757568880130579 [ 6948.346184] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6948.349330] LustreError: 232224:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6948.494763] Lustre: server umount lustre-MDT0001 complete [ 7054.018289] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7055.129761] Key type lgssc unregistered [ 7055.260329] LNet: 235478:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7056.293375] LNet: Removed LNI 192.168.201.107@tcp [ 7056.614770] Key type .llcrypt unregistered [ 7056.616603] Key type ._llcrypt unregistered [ 7065.315110] Key type ._llcrypt registered [ 7065.316464] Key type .llcrypt registered [ 7065.361639] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 7071.835114] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7072.186177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7072.248073] alg: No test for adler32 (adler32-zlib) [ 7073.128051] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 7073.220194] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 7073.222465] LNet: Accept secure, port 988 [ 7074.809814] Key type lgssc registered [ 7075.232664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7078.860493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7081.284175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7083.611402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7086.104992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7092.430093] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7095.995196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7096.016782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7096.021302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7097.103836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7097.116987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7097.156925] Lustre: lustre-MDT0000: new disk, initializing [ 7097.191385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7097.199067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7098.646904] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7103.054344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7103.077856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7103.108926] Lustre: 239448:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7103.131873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7103.134110] Lustre: Skipped 1 previous similar message [ 7103.172927] Lustre: lustre-MDT0001: new disk, initializing [ 7103.205181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7103.218339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7103.221822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7104.618959] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7107.012531] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7109.521095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7109.547700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7109.641920] Lustre: lustre-OST0000: new disk, initializing [ 7109.644506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7109.675672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7111.048953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7111.053572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7111.111734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7111.852211] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7116.111290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7116.143309] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7116.208757] Lustre: lustre-OST0001: new disk, initializing [ 7116.211159] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7116.250478] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7118.213617] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7119.038817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7119.042709] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7119.071024] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7123.830726] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7128.711894] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7139.296877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7139.298953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.304168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7139.811807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.816274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7139.819388] Lustre: Skipped 2 previous similar messages [ 7142.990302] LustreError: 242944:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7143.024788] Lustre: server umount lustre-MDT0000 complete [ 7144.226061] LustreError: 240223:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749521 with bad export cookie 11971559990671021938 [ 7144.229089] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7144.229923] LustreError: 240223:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7144.291914] LustreError: 243146:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7144.293926] LustreError: 243146:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7144.366226] Lustre: server umount lustre-MDT0001 complete [ 7155.956124] LustreError: 243346:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 7155.958502] LustreError: 243346:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7155.978562] Lustre: server umount lustre-OST0000 complete [ 7167.540574] LustreError: 243548:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7167.545619] LustreError: 243548:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7167.617250] Lustre: server umount lustre-OST0001 complete [ 7172.420516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7175.359862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7175.701025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7178.560811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7181.888866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7182.307612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7189.292069] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7193.050970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7193.056854] Lustre: lustre-MDT0000: reset Object Index mappings [ 7193.225564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7194.597296] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7197.731038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7197.736248] Lustre: lustre-MDT0001: reset Object Index mappings [ 7199.638815] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7200.855804] Lustre: 247941:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7203.684812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7203.829702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7203.832670] Lustre: Skipped 1 previous similar message [ 7206.000852] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7207.909729] LustreError: 248296:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7207.918095] LustreError: 248296:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 7209.645632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7211.842487] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7218.217389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7218.218358] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7225.821957] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7227.137973] Lustre: 249794:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7234.017039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7234.021923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.027755] Lustre: Skipped 2 previous similar messages [ 7234.030560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7238.627772] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7238.628135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7238.631448] Lustre: Skipped 2 previous similar messages [ 7238.633302] Lustre: Skipped 2 previous similar messages [ 7239.685814] LustreError: 249899:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7239.687596] LustreError: 249899:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7239.727167] Lustre: server umount lustre-MDT0000 complete [ 7241.102160] LustreError: 246826:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749618 with bad export cookie 11971559990671028658 [ 7241.105640] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7241.105956] LustreError: 246826:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7241.254601] Lustre: server umount lustre-MDT0001 complete [ 7252.983407] LustreError: 250303:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 7252.986317] LustreError: 250303:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 7253.019171] Lustre: server umount lustre-OST0000 complete [ 7264.690060] Lustre: server umount lustre-OST0001 complete [ 7271.300109] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 20:40:48 (1753749648) [ 7276.826931] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 7280.090702] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7283.791531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7286.191131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7288.394266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7290.712946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7293.173626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7293.194576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7293.275789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7293.291074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7293.330461] Lustre: lustre-MDT0000: new disk, initializing [ 7293.355913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7293.357863] Lustre: Skipped 1 previous similar message [ 7293.364336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7294.721885] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7299.017391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7299.039567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7299.070577] Lustre: 254006:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7299.166585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7300.514635] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7303.590577] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7304.932480] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7307.268245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7307.292634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7307.393069] Lustre: lustre-OST0000: new disk, initializing [ 7307.395033] Lustre: Skipped 1 previous similar message [ 7307.397511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7307.400413] Lustre: Skipped 2 previous similar messages [ 7308.802787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7308.804983] Lustre: Skipped 1 previous similar message [ 7308.806650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7308.839174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7309.475918] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7312.699096] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7314.105235] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7314.195444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7315.593322] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7315.679254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7318.447086] LustreError: 256914:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7318.449221] LustreError: 256914:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7318.510364] Lustre: server umount lustre-OST0000 complete [ 7324.129616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7324.130555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7324.134665] Lustre: Skipped 1 previous similar message [ 7324.142598] Lustre: Skipped 1 previous similar message [ 7334.368098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7334.369445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7334.372658] Lustre: Skipped 3 previous similar messages [ 7334.423974] Lustre: server umount lustre-MDT0000 complete [ 7335.393147] Lustre: 236983:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753749696/real 1753749696] req@ffff9a6801aaf480 x1838939583415936/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1753749712 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7335.393401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7335.403258] Lustre: 236983:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7335.407883] LustreError: 254798:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7335.414603] LustreError: 254798:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7335.698406] LustreError: 253999:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749713 with bad export cookie 11971559990671030751 [ 7335.701724] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7335.703123] LustreError: 253999:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7335.875638] Lustre: server umount lustre-MDT0001 complete [ 7342.441083] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7345.964614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7346.111456] LustreError: 258480:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7346.127926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7346.130697] Lustre: Skipped 2 previous similar messages [ 7347.469832] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7350.459284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7351.905512] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7352.933752] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7355.269162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7357.442202] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7360.419577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7360.444524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7360.498775] Lustre: lustre-OST0001: new disk, initializing [ 7360.500417] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7362.118676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7362.122718] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7362.156203] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7362.552949] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7367.297252] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7368.680186] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7375.629739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7375.634066] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7377.890271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7377.891529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7377.895159] Lustre: Skipped 3 previous similar messages [ 7377.899361] Lustre: Skipped 3 previous similar messages [ 7383.677585] LustreError: 261959:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7383.679621] LustreError: 261959:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7383.711235] Lustre: server umount lustre-MDT0000 complete [ 7384.873164] LustreError: 258461:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749762 with bad export cookie 11971559990671031633 [ 7384.876538] LustreError: 258461:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7384.876762] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7385.014031] Lustre: server umount lustre-MDT0001 complete [ 7396.353264] Lustre: server umount lustre-OST0000 complete [ 7406.841558] Lustre: server umount lustre-OST0001 complete [ 7412.312256] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7415.666560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7415.809670] LustreError: 263726:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7415.832742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7415.834653] Lustre: Skipped 3 previous similar messages [ 7417.087131] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7419.936857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7421.455672] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7422.415158] Lustre: 264823:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7424.606285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7425.763449] LustreError: 265363:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7425.771498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7426.835059] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7429.924979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7431.158597] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7432.145901] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7435.496021] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7436.702660] Lustre: 266661:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7443.455859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7443.472357] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7446.652796] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 20:43:43 (1753749823) [ 7449.724033] LustreError: 267349:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7449.725776] LustreError: 267349:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 7449.753274] Lustre: server umount lustre-MDT0000 complete [ 7450.595236] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7450.596810] LustreError: 266428:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7450.600178] Lustre: Skipped 1 previous similar message [ 7450.603720] LustreError: 266428:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 7450.959964] LustreError: 266663:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749828 with bad export cookie 11971559990671037919 [ 7450.961981] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7450.964143] LustreError: 266663:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7455.714530] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7455.716962] Lustre: Skipped 5 previous similar messages [ 7457.176562] Lustre: server umount lustre-MDT0001 complete [ 7458.498732] Lustre: server umount lustre-OST0000 complete [ 7459.792429] Lustre: server umount lustre-OST0001 complete [ 7465.416593] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 7468.644733] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7472.033864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7474.492708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7476.533166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7478.725475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7481.328461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7481.346762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7481.422723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7481.436139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7481.471878] Lustre: lustre-MDT0000: new disk, initializing [ 7481.500557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7482.763216] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7487.016048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7487.038585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7487.071368] Lustre: 270774:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7487.168581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7488.671316] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7491.744974] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7493.073253] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7495.403660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7495.425844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7496.744402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7497.311143] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7500.300551] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7501.024470] Lustre: Mounted lustre-client [ 7501.304834] LustreError: 273260:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff9a6802552800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 7501.326133] Lustre: Unmounted lustre-client [ 7506.287534] Lustre: server umount lustre-OST0000 complete [ 7507.937085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7517.695924] Lustre: server umount lustre-MDT0000 complete [ 7519.127223] LustreError: 270764:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749896 with bad export cookie 11971559990671045332 [ 7519.129934] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7519.131835] LustreError: 270764:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7529.516381] Lustre: server umount lustre-MDT0001 complete [ 7534.131213] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7535.130180] Key type lgssc unregistered [ 7562.912387] LNet: 274533:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7632.550523] LNet: Removed LNI 192.168.201.107@tcp [ 7632.938621] Key type .llcrypt unregistered [ 7632.940250] Key type ._llcrypt unregistered [ 7638.997259] Key type ._llcrypt registered [ 7638.998396] Key type .llcrypt registered [ 7639.032180] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 7645.981949] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7646.503923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7646.521961] alg: No test for adler32 (adler32-zlib) [ 7647.402662] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 7647.484686] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 7647.485933] LNet: Accept secure, port 988 [ 7649.072112] Key type lgssc registered [ 7649.418574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7652.209936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7654.590871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7656.914192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7659.310798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7659.918177] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 20:47:17 (1753750037) [ 7678.405566] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 20:47:35 (1753750055) [ 7694.327497] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7695.394311] Key type lgssc unregistered [ 7695.534567] LNet: 280229:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7695.544319] LNet: Removed LNI 192.168.201.107@tcp [ 7695.863553] Key type .llcrypt unregistered [ 7695.865459] Key type ._llcrypt unregistered [ 7722.720115] Key type ._llcrypt registered [ 7722.721345] Key type .llcrypt registered [ 7722.761368] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7722.873279] Key type .llcrypt unregistered [ 7722.874503] Key type ._llcrypt unregistered [ 7733.774297] Key type ._llcrypt registered [ 7733.775655] Key type .llcrypt registered [ 7733.816298] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7733.931487] Key type .llcrypt unregistered [ 7733.932868] Key type ._llcrypt unregistered [ 7761.822442] Key type ._llcrypt registered [ 7761.823651] Key type .llcrypt registered [ 7761.865161] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7761.975332] Key type .llcrypt unregistered [ 7761.976407] Key type ._llcrypt unregistered [ 7772.974952] Key type ._llcrypt registered [ 7772.976387] Key type .llcrypt registered [ 7773.018651] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7773.138414] Key type .llcrypt unregistered [ 7773.140057] Key type ._llcrypt unregistered [ 7784.090840] Key type ._llcrypt registered [ 7784.092080] Key type .llcrypt registered [ 7784.132772] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7784.246295] Key type .llcrypt unregistered [ 7784.247667] Key type ._llcrypt unregistered [ 7812.207490] Key type ._llcrypt registered [ 7812.208460] Key type .llcrypt registered [ 7812.249852] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7812.380537] Key type .llcrypt unregistered [ 7812.381983] Key type ._llcrypt unregistered [ 7823.682764] Key type ._llcrypt registered [ 7823.683946] Key type .llcrypt registered [ 7823.725655] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7823.840650] Key type .llcrypt unregistered [ 7823.842091] Key type ._llcrypt unregistered [ 7834.786544] Key type ._llcrypt registered [ 7834.787689] Key type .llcrypt registered [ 7834.818633] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7834.920317] Key type .llcrypt unregistered [ 7834.921700] Key type ._llcrypt unregistered [ 7853.727819] Key type ._llcrypt registered [ 7853.729354] Key type .llcrypt registered [ 7853.773295] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 7859.884859] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7860.192789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7860.249654] alg: No test for adler32 (adler32-zlib) [ 7861.090476] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 7861.166555] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 7861.168205] LNet: Accept secure, port 988 [ 7862.744157] Key type lgssc registered [ 7863.141294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7866.844277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7869.977668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7872.267580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7874.666860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7875.316220] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7630 sec ========== 20:50:52 (1753750252) [ 7875.859937] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:50:53 (1753750253) === [ 7877.035889] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:50:54 (1753750254) === [ 7890.548482] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7891.677824] Key type lgssc unregistered [ 7891.814504] LNet: 296024:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7891.822463] LNet: Removed LNI 192.168.201.107@tcp [ 7892.142490] Key type .llcrypt unregistered [ 7892.143874] Key type ._llcrypt unregistered