[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 458523355 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003326] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007898] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010249] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012044] SELinux: Initializing. [ 0.013068] *** VALIDATE selinux *** [ 0.021236] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025314] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026127] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027087] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028090] *** VALIDATE tmpfs *** [ 0.029323] *** VALIDATE proc *** [ 0.030167] *** VALIDATE cgroup *** [ 0.031007] *** VALIDATE cgroup2 *** [ 0.032201] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033118] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035023] Spectre V2 : User space: Vulnerable [ 0.036019] Speculative Store Bypass: Vulnerable [ 0.040388] debug: unmapping init [mem 0xffffffffb4a59000-0xffffffffb4a60fff] [ 0.042150] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043726] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044026] ... version: 2 [ 0.045018] ... bit width: 48 [ 0.046015] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048021] ... max period: 00007fffffffffff [ 0.049016] ... fixed-purpose events: 3 [ 0.050014] ... event mask: 000000070000000f [ 0.051333] rcu: Hierarchical SRCU implementation. [ 0.053449] smp: Bringing up secondary CPUs ... [ 0.054567] x86: Booting SMP configuration: [ 0.055032] .... node #0, CPUs: #1 #2 #3 [ 0.058348] smp: Brought up 1 node, 4 CPUs [ 0.060015] smpboot: Max logical packages: 1 [ 0.061019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135517] node 0 deferred pages initialised in 71ms [ 0.138248] devtmpfs: initialized [ 0.139232] x86/mm: Memory block size: 128MB [ 0.141122] gcov: version magic: 0x41383552 [ 0.143176] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149289] pinctrl core: initialized pinctrl subsystem [ 0.151186] [ 0.151787] ************************************************************* [ 0.154015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157015] ** ** [ 0.159010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162015] ** ** [ 0.164012] ** This means that this kernel is built to expose internal ** [ 0.167015] ** IOMMU data structures, which may compromise security on ** [ 0.168011] ** your system. ** [ 0.171015] ** ** [ 0.173012] ** If you see this message and you are not debugging the ** [ 0.176012] ** kernel, report this immediately to your vendor! ** [ 0.178016] ** ** [ 0.181012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183054] ************************************************************* [ 0.186889] NET: Registered protocol family 16 [ 0.189437] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.194064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198125] cpuidle: using governor menu [ 0.199817] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.200503] PCI: Using configuration type 1 for base access [ 0.201127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.206150] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.208153] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.211067] cryptd: max_cpu_qlen set to 1000 [ 0.213397] ACPI: Added _OSI(Module Device) [ 0.215019] ACPI: Added _OSI(Processor Device) [ 0.216023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.218019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.222387] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.228429] ACPI: Interpreter enabled [ 0.230082] ACPI: PM: (supports S0 S3 S4 S5) [ 0.231017] ACPI: Using IOAPIC for interrupt routing [ 0.233139] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.237441] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.248000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.250063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.253032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.256193] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.261748] acpiphp: Slot [2] registered [ 0.263209] acpiphp: Slot [5] registered [ 0.265198] acpiphp: Slot [6] registered [ 0.266141] acpiphp: Slot [7] registered [ 0.268139] acpiphp: Slot [8] registered [ 0.269135] acpiphp: Slot [9] registered [ 0.271171] acpiphp: Slot [10] registered [ 0.272143] acpiphp: Slot [3] registered [ 0.273120] acpiphp: Slot [4] registered [ 0.275190] acpiphp: Slot [11] registered [ 0.276153] acpiphp: Slot [12] registered [ 0.277147] acpiphp: Slot [13] registered [ 0.279149] acpiphp: Slot [14] registered [ 0.280202] acpiphp: Slot [15] registered [ 0.282112] acpiphp: Slot [16] registered [ 0.284122] acpiphp: Slot [17] registered [ 0.285112] acpiphp: Slot [18] registered [ 0.287080] acpiphp: Slot [19] registered [ 0.288061] acpiphp: Slot [20] registered [ 0.289060] acpiphp: Slot [21] registered [ 0.290070] acpiphp: Slot [22] registered [ 0.291138] acpiphp: Slot [23] registered [ 0.293106] acpiphp: Slot [24] registered [ 0.294104] acpiphp: Slot [25] registered [ 0.295117] acpiphp: Slot [26] registered [ 0.297098] acpiphp: Slot [27] registered [ 0.298083] acpiphp: Slot [28] registered [ 0.299063] acpiphp: Slot [29] registered [ 0.301071] acpiphp: Slot [30] registered [ 0.302119] acpiphp: Slot [31] registered [ 0.303054] PCI host bridge to bus 0000:00 [ 0.304016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.306025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.309022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.310016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.312019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.314023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.316137] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.317748] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.319985] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.328079] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.332108] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.334023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.337016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.338015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.340660] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.344724] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.347045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.350928] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.357019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.372917] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.378014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.384561] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.392017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.399019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.415017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.423070] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.435019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.439017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.453013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.460012] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.466014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.471012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.482016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.490397] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.496022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.502013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.514015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.522759] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.529017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.541024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.554026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.564663] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.571019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.578018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.588026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.597892] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.600432] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.602410] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.605397] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.607224] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.614050] iommu: Default domain type: Passthrough [ 0.616498] SCSI subsystem initialized [ 0.618173] ACPI: bus type USB registered [ 0.619112] usbcore: registered new interface driver usbfs [ 0.621062] usbcore: registered new interface driver hub [ 0.622073] usbcore: registered new device driver usb [ 0.624167] pps_core: LinuxPPS API ver. 1 registered [ 0.626014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.628085] PTP clock support registered [ 0.630154] EDAC MC: Ver: 3.0.0 [ 0.632229] PCI: Using ACPI for IRQ routing [ 0.633835] NetLabel: Initializing [ 0.635010] NetLabel: domain hash size = 128 [ 0.636011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.638111] NetLabel: unlabeled traffic allowed by default [ 0.640198] vgaarb: loaded [ 0.641301] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.644016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.649407] clocksource: Switched to clocksource kvm-clock [ 0.761720] VFS: Disk quotas dquot_6.6.0 [ 0.763277] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.765774] *** VALIDATE ramfs *** [ 0.766891] *** VALIDATE hugetlbfs *** [ 0.768536] pnp: PnP ACPI init [ 0.770882] pnp: PnP ACPI: found 6 devices [ 0.787860] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.791165] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.793380] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.795740] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.798372] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.800807] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.803845] NET: Registered protocol family 2 [ 0.806462] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.811662] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.816060] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.823080] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.826860] TCP: Hash tables configured (established 65536 bind 65536) [ 0.829188] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.832713] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.835963] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.839351] NET: Registered protocol family 1 [ 0.842568] RPC: Registered named UNIX socket transport module. [ 0.844381] RPC: Registered udp transport module. [ 0.845883] RPC: Registered tcp transport module. [ 0.847850] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.850447] NET: Registered protocol family 44 [ 0.851952] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.853580] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.855321] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.857400] PCI: CLS 0 bytes, default 64 [ 0.859771] Unpacking initramfs... [ 2.247488] debug: unmapping init [mem 0xffff8c7ebcc54000-0xffff8c7ebffbffff] [ 2.250963] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.253160] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.255843] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.681669] Initialise system trusted keyrings [ 2.682871] Key type blacklist registered [ 2.684333] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.690804] zbud: loaded [ 2.692961] *** VALIDATE nfs *** [ 2.693927] *** VALIDATE nfs4 *** [ 2.695383] pstore: using deflate compression [ 2.698081] Platform Keyring initialized [ 2.773763] NET: Registered protocol family 38 [ 2.775403] Key type asymmetric registered [ 2.776387] Asymmetric key parser 'x509' registered [ 2.777649] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.779817] io scheduler mq-deadline registered [ 2.781382] io scheduler kyber registered [ 2.782454] io scheduler bfq registered [ 2.783756] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.785893] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.787853] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.789877] ACPI: Power Button [PWRF] [ 2.793515] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.798697] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.807968] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.817968] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.852544] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.880233] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.909835] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.915364] Non-volatile memory driver v1.3 [ 2.916675] Linux agpgart interface v0.103 [ 2.952529] virtio_blk virtio1: [vda] 146696 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.955725] vda: detected capacity change from 0 to 75108352 [ 2.981555] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.985655] vdb: detected capacity change from 0 to 1073741824 [ 3.007717] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.010881] vdc: detected capacity change from 0 to 2621440000 [ 3.026151] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.029458] vdd: detected capacity change from 0 to 2621440000 [ 3.044160] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.046659] vde: detected capacity change from 0 to 4294967296 [ 3.058255] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.059971] vdf: detected capacity change from 0 to 4294967296 [ 3.064574] libphy: Fixed MDIO Bus: probed [ 3.069084] usbcore: registered new interface driver usbserial_generic [ 3.071800] usbserial: USB Serial support registered for generic [ 3.074418] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.079599] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.081702] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.084406] mousedev: PS/2 mouse device common for all mice [ 3.087485] rtc_cmos 00:05: RTC can wake from S4 [ 3.090228] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.090917] rtc_cmos 00:05: registered as rtc0 [ 3.095833] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.096901] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.098942] intel_pstate: CPU model not supported [ 3.104153] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.106886] hid: raw HID events driver (C) Jiri Kosina [ 3.109395] usbcore: registered new interface driver usbhid [ 3.111478] usbhid: USB HID core driver [ 3.112987] drop_monitor: Initializing network drop monitor service [ 3.115533] Initializing XFRM netlink socket [ 3.117545] NET: Registered protocol family 10 [ 3.120446] Segment Routing with IPv6 [ 3.121904] NET: Registered protocol family 17 [ 3.123863] mpls_gso: MPLS GSO support [ 3.130304] RAS: Correctable Errors collector initialized. [ 3.132715] AVX version of gcm_enc/dec engaged. [ 3.134556] AES CTR mode by8 optimization enabled [ 3.248040] sched_clock: Marking stable (3247997080, 0)->(4171864556, -923867476) [ 3.251545] registered taskstats version 1 [ 3.253586] Loading compiled-in X.509 certificates [ 3.255759] zswap: loaded using pool lzo/zbud [ 3.282616] Key type big_key registered [ 3.296744] Key type encrypted registered [ 3.298515] ima: No TPM chip found, activating TPM-bypass! [ 3.300746] ima: Allocated hash algorithm: sha1 [ 3.302511] ima: No architecture policies found [ 3.304721] evm: Initialising EVM extended attributes: [ 3.306624] evm: security.selinux [ 3.307918] evm: security.ima [ 3.309118] evm: security.capability [ 3.310525] evm: HMAC attrs: 0x1 [ 3.313184] rtc_cmos 00:05: setting system clock to 2026-09-13 10:55:40 UTC (1789296940) [ 3.319151] debug: unmapping init [mem 0xffffffffb5a03000-0xffffffffb5bfffff] [ 3.322355] debug: unmapping init [mem 0xffffffffb4782000-0xffffffffb4a58fff] [ 3.331140] Write protecting the kernel read-only data: 28672k [ 3.334267] debug: unmapping init [mem 0xffffffffb2e03000-0xffffffffb2ffffff] [ 3.337367] debug: unmapping init [mem 0xffffffffb3714000-0xffffffffb37fffff] [ 3.364980] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.371054] systemd[1]: Detected virtualization kvm. [ 3.372645] systemd[1]: Detected architecture x86-64. [ 3.373914] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.402152] systemd[1]: No hostname configured. [ 3.403663] systemd[1]: Set hostname to . [ 3.405372] random: systemd: uninitialized urandom read (16 bytes read) [ 3.407951] systemd[1]: Initializing machine ID from random generator. [ 3.459910] random: ln: uninitialized urandom read (6 bytes read) [ 3.547926] random: systemd: uninitialized urandom read (16 bytes read) [ 3.550514] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.555778] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.561514] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.173482] device-mapper: uevent: version 1.0.3 [ 4.175375] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] [ 4.843502] random: fast init done Started Hardware RNG Entropy Gatherer Daemon. [ 4.929880] virtio_net virtio0 ens2: renamed from eth0 [ 5.024234] scsi host0: ata_piix [ 5.043262] scsi host1: ata_piix [ 5.044555] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.047050] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.689235] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.817521] random: crng init done [ 9.819135] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.168822] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.284497] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.531590] SELinux: Disabled at runtime. [ 11.594907] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.603986] systemd[1]: Detected virtualization kvm. [ 11.605972] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.107924] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.111343] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.116300] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.122317] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.126120] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.133591] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.139836] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.177396] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.573338] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.848661] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.888870] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.998429] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.015038] EDAC sbridge: Ver: 1.1.2 [ 14.346843] Key type dns_resolver registered [ 14.640522] NFS: Registering the id_resolver key type [ 14.642561] Key type id_resolver registered [ 14.643689] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg237-server login: [ 55.716777] hrtimer: interrupt took 5048915 ns [ 56.577074] libcfs: loading out-of-tree module taints kernel. [ 56.625303] Key type ._llcrypt registered [ 56.628532] Key type .llcrypt registered [ 56.841603] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 80.892243] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 82.753158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.771079] alg: No test for adler32 (adler32-zlib) [ 84.643338] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 85.596837] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 87.448467] Key type lgssc registered [ 89.517242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 112.028282] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 159.969324] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 174.953376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 175.013178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.626975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 176.695365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 176.923371] Lustre: lustre-MDT0000: new disk, initializing [ 177.120768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.141889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 182.640140] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.982847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.209807] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 199.253655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 199.261562] Lustre: Skipped 1 previous similar message [ 199.402617] Lustre: lustre-MDT0001: new disk, initializing [ 199.481721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.540284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 199.564324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 205.085081] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.874642] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 222.176429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.507472] Lustre: lustre-OST0000: new disk, initializing [ 222.510612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 222.516639] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.583639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.670562] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.442786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 231.447480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.490585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 245.610584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.817234] Lustre: lustre-OST0001: new disk, initializing [ 245.828350] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 245.835584] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.935731] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 253.147161] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.606078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 253.633946] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 253.710324] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 266.992257] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 279.360548] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 287.196411] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing check_logdir /tmp/testlogs/ [ 293.998302] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing yml_node [ 299.001993] Lustre: DEBUG MARKER: Client: 2.17.57.108 [ 301.475939] Lustre: DEBUG MARKER: MDS: 2.17.57.108 [ 304.147528] Lustre: DEBUG MARKER: OSS: 2.17.57.108 [ 306.030730] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 13 07:00:40 EDT 2026 [ 327.062346] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 329.289735] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 346.081435] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.084211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.090227] Lustre: Skipped 3 previous similar messages [ 346.107821] Lustre: Skipped 3 previous similar messages [ 351.201764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.210169] Lustre: Skipped 3 previous similar messages [ 351.872666] Lustre: server umount lustre-MDT0000 complete [ 355.655308] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297292 with bad export cookie 14966133072143904753 [ 355.658778] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 355.669017] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 356.321188] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.340231] Lustre: Skipped 1 previous similar message [ 356.340965] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 356.352354] Lustre: Skipped 1 previous similar message [ 360.353216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 360.359096] Lustre: Skipped 1 previous similar message [ 362.036697] Lustre: server umount lustre-MDT0001 complete [ 365.807259] Lustre: server umount lustre-OST0000 complete [ 369.131735] Lustre: server umount lustre-OST0001 complete [ 375.718532] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 385.344107] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 422.036249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.291859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 422.322775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 422.424774] Lustre: lustre-MDT0000: new disk, initializing [ 422.532890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.548751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 427.121154] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 438.202950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.302967] Lustre: 15091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 438.370945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 438.379574] Lustre: Skipped 1 previous similar message [ 438.455384] Lustre: lustre-MDT0001: new disk, initializing [ 438.557265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 438.607179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 438.629382] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 443.685691] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 453.656486] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 459.411769] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 466.588726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 466.992373] Lustre: lustre-OST0000: new disk, initializing [ 466.998127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 467.004390] Lustre: 17061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 467.132314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.415760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 468.422153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 468.595884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 473.602652] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 483.481525] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 487.359966] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 487.592707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 492.494476] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 492.682621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 499.169129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 499.171658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.176748] LustreError: Skipped 1 previous similar message [ 499.194077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 499.204028] Lustre: Skipped 1 previous similar message [ 499.220083] Lustre: Skipped 1 previous similar message [ 500.989952] Lustre: server umount lustre-OST0000 complete [ 505.313636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 505.323957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.409230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.410412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 509.435158] Lustre: Skipped 1 previous similar message [ 511.000168] Lustre: server umount lustre-MDT0000 complete [ 514.530785] LustreError: 15102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.569903] LustreError: 15102:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 514.715430] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297451 with bad export cookie 14966133072143906286 [ 514.731960] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 514.736132] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 515.170263] Lustre: server umount lustre-MDT0001 complete [ 522.691690] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 07:04:16 (1789297456) [ 542.324178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 543.026377] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.116790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.831324] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 548.321124] LustreError: 19935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.451800] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.900486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.347440] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 573.621471] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 580.954896] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 583.364780] Lustre: Failing over lustre-MDT0000 [ 583.382204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.398059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 583.763195] Lustre: server umount lustre-MDT0000 complete [ 588.259232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 588.278749] LustreError: 19935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.011277] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.031304] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 604.129340] Lustre: 3639:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789297525/real 1789297525] req@ffff8c7f3d995c00 x1876213914308864/t0(0) o400->MGC192.168.202.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789297541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 604.169190] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 604.313806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.693815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.698979] Lustre: Skipped 1 previous similar message [ 614.770527] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 619.860157] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 620.008800] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 620.019122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 620.047285] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 629.810138] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 635.362575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 635.374565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.403245] Lustre: Skipped 1 previous similar message [ 635.418429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 640.406570] Lustre: server umount lustre-MDT0000 complete [ 642.532598] LustreError: 19935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.552175] LustreError: 19935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 644.111484] LustreError: 19916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297581 with bad export cookie 14966133072143907700 [ 644.112750] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.119644] LustreError: 19916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 644.656654] Lustre: server umount lustre-MDT0001 complete [ 653.737237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.604684] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 671.883708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.308309] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 686.195079] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 692.332097] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 694.694076] Lustre: Failing over lustre-MDT0000 [ 695.083269] Lustre: server umount lustre-MDT0000 complete [ 697.825188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 697.828605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.843123] Lustre: Skipped 1 previous similar message [ 697.849985] LustreError: 24001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 697.868342] LustreError: 24001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 714.212204] Lustre: 3637:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789297635/real 1789297635] req@ffff8c7e07a55500 x1876213914338688/t0(0) o400->MGC192.168.202.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789297651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 714.247808] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 715.601797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 725.877981] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 730.657697] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 731.109089] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 731.113038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 731.134207] Lustre: Skipped 1 previous similar message [ 731.156104] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 741.327559] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 746.465510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 746.476335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 746.497590] Lustre: Skipped 2 previous similar messages [ 751.816323] Lustre: server umount lustre-MDT0000 complete [ 755.483644] LustreError: 23253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297692 with bad export cookie 14966133072143908638 [ 755.492180] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 755.502214] LustreError: 23253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 755.951595] Lustre: server umount lustre-MDT0001 complete [ 767.305321] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 07:08:21 (1789297701) [ 787.956784] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 803.503470] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 847.920729] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 858.981628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 859.192300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 859.214475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 859.272730] Lustre: lustre-MDT0000: new disk, initializing [ 859.337331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.342411] Lustre: Skipped 3 previous similar messages [ 859.354395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 863.657885] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 874.620618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 874.743251] Lustre: 29673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 874.983414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 880.170477] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 891.354601] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 898.501340] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 905.587295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 905.814285] Lustre: 31643:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 907.169927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 907.225281] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 907.273570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 907.289564] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 912.368814] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 922.394800] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 928.497119] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 953.326791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 953.330369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.339776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.339788] Lustre: Skipped 2 previous similar messages [ 953.373158] Lustre: Skipped 4 previous similar messages [ 956.276842] Lustre: server umount lustre-OST0000 complete [ 966.870626] Lustre: server umount lustre-MDT0000 complete [ 968.674721] LustreError: 29678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.711326] LustreError: 29678:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 970.788956] LustreError: 29666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297907 with bad export cookie 14966133072143909044 [ 970.806051] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 970.824850] LustreError: 29666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 971.250757] Lustre: server umount lustre-MDT0001 complete [ 982.514607] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 986.241768] Key type lgssc unregistered [ 986.615789] LNet: 33620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.651583] LNetError: 33620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.697087] LNet: Removed LNI 192.168.202.137@tcp [ 989.280371] Key type .llcrypt unregistered [ 989.283934] Key type ._llcrypt unregistered [ 1018.068723] Key type ._llcrypt registered [ 1018.071388] Key type .llcrypt registered [ 1018.173970] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1019.195836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.250069] alg: No test for adler32 (adler32-zlib) [ 1020.323966] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 1020.518076] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1022.160255] Key type lgssc registered [ 1023.231872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1030.786349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1030.812405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.371107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1037.474734] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1041.890153] LustreError: 35034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.899612] LustreError: 35034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1047.022851] LustreError: 35035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1047.158394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.682450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1053.462337] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1061.373551] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1067.459832] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1074.718789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1075.136822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1077.248564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1081.460733] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1089.042293] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1092.622415] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1106.015179] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 07:14:00 (1789298040) [ 1108.188444] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1110.039953] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1116.964803] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 07:14:11 (1789298051) [ 1128.153391] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 07:14:22 (1789298062) [ 1129.393485] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1139.457487] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1139.466037] Lustre: Skipped 2 previous similar messages [ 1141.274294] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 07:14:36 (1789298076) [ 1143.696503] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1143.703820] Lustre: Skipped 2 previous similar messages [ 1152.573735] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1152.579174] Lustre: Skipped 4 previous similar messages [ 1165.789468] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 07:15:00 (1789298100) [ 1168.540891] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1172.566640] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1180.799968] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 07:15:15 (1789298115) [ 1182.696560] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1182.700394] Lustre: Skipped 6 previous similar messages [ 1195.332386] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 07:15:30 (1789298130) [ 1215.475936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1215.477158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.503078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1215.512720] Lustre: Skipped 1 previous similar message [ 1215.524059] Lustre: Skipped 1 previous similar message [ 1216.700947] Lustre: server umount lustre-OST0000 complete [ 1221.602274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1221.611072] LustreError: Skipped 1 previous similar message [ 1221.615532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1221.636659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.698505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.700201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.575850] Lustre: server umount lustre-MDT0000 complete [ 1230.819981] LustreError: 35035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.846943] LustreError: 35035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1231.261768] LustreError: 35014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298168 with bad export cookie 6339949601034150137 [ 1231.264398] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1231.277022] LustreError: 35014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1231.666720] Lustre: server umount lustre-MDT0001 complete [ 1245.074584] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1248.436343] Key type lgssc unregistered [ 1248.827798] LNet: 42902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.835753] LNetError: 42902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.855053] LNet: Removed LNI 192.168.202.137@tcp [ 1250.004945] Key type .llcrypt unregistered [ 1250.011276] Key type ._llcrypt unregistered [ 1270.362635] Key type ._llcrypt registered [ 1270.364507] Key type .llcrypt registered [ 1270.475739] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 1289.161840] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1290.171769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1290.292586] alg: No test for adler32 (adler32-zlib) [ 1291.287206] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 1291.557181] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1293.248175] Key type lgssc registered [ 1294.482288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1332.498950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1332.555952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.803945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1333.843129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1333.991573] Lustre: lustre-MDT0000: new disk, initializing [ 1334.146796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.233720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1339.552813] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1352.081721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.254716] Lustre: 46052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1352.326522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1352.333161] Lustre: Skipped 1 previous similar message [ 1352.436514] Lustre: lustre-MDT0001: new disk, initializing [ 1352.581509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1352.644053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1352.662038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1357.204930] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1367.279435] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1374.679699] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1382.853869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1383.210420] Lustre: lustre-OST0000: new disk, initializing [ 1383.214750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1383.223760] Lustre: 48024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1383.320756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1390.232131] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1391.275098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1391.285195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1391.517415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1400.574369] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1405.726947] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1406.062363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1411.107465] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1411.361624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1417.185130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1417.186735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.196171] LustreError: Skipped 1 previous similar message [ 1417.197040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1417.246238] Lustre: Skipped 1 previous similar message [ 1419.739232] Lustre: server umount lustre-OST0000 complete [ 1424.359607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1424.365187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.380218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1424.388327] Lustre: Skipped 1 previous similar message [ 1427.427358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1427.428184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1430.361081] Lustre: server umount lustre-MDT0000 complete [ 1434.352901] LustreError: 46046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298371 with bad export cookie 1089337492319134691 [ 1434.353961] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1434.371069] LustreError: 46046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1434.888828] Lustre: server umount lustre-MDT0001 complete [ 1446.926673] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 07:19:41 (1789298381) [ 1455.068115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1455.648421] LustreError: 50485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.671623] LustreError: 50485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1455.741716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.235643] LustreError: 50486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.815914] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1465.323975] LustreError: 50485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1470.434647] LustreError: 50486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1472.761028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.409820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1480.500742] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1488.900588] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1496.414747] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1504.155763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1504.529181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1510.755564] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1518.537807] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1528.784159] LustreError: 53017:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1529.730902] Lustre: Modifying parameter general.timeout=1 in log params [ 1531.504583] Lustre: Modifying parameter general.timeout=3 in log params [ 1531.511299] Lustre: Skipped 1 previous similar message [ 1533.638412] Lustre: Modifying parameter general.timeout=5 in log params [ 1533.642442] Lustre: Skipped 1 previous similar message [ 1538.220095] Lustre: Modifying parameter general.timeout=10 in log params [ 1538.224109] Lustre: Skipped 4 previous similar messages [ 1546.982911] Lustre: Modifying parameter general.timeout=18 in log params [ 1546.985714] Lustre: Skipped 7 previous similar messages [ 1563.811625] Lustre: Modifying parameter general.timeout=32 in log params [ 1563.814177] Lustre: Skipped 13 previous similar messages [ 1596.524131] Lustre: Modifying parameter general.timeout=63 in log params [ 1596.533820] Lustre: Skipped 30 previous similar messages [ 1661.184220] Lustre: Modifying parameter general.timeout=127 in log params [ 1661.192098] Lustre: Skipped 63 previous similar messages [ 1674.985501] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 07:23:29 (1789298609) [ 1678.054609] Lustre: 59971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1679.961387] Lustre: 60070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1682.223255] Lustre: 60168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1687.903633] Lustre: 60486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1687.910586] Lustre: 60486:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1714.893267] Lustre: 61839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1714.916394] LustreError: 61839:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1719.036401] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 07:24:13 (1789298653) [ 1732.578902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.587975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1732.605136] Lustre: Skipped 1 previous similar message [ 1732.634499] Lustre: Skipped 3 previous similar messages [ 1737.699213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1737.708535] Lustre: Skipped 2 previous similar messages [ 1737.980951] Lustre: server umount lustre-MDT0000 complete [ 1742.053165] LustreError: 50466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298679 with bad export cookie 1089337492319135573 [ 1742.065432] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1742.382154] Lustre: server umount lustre-MDT0001 complete [ 1756.564785] Lustre: server umount lustre-OST0000 complete [ 1785.248898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.531537] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1785.842350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.090788] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1800.521733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.600241] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1800.625964] Lustre: 64371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1806.694699] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1813.870750] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1819.734407] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1826.705501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1826.960232] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1827.330760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1827.343542] Lustre: Skipped 1 previous similar message [ 1833.381973] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1840.048391] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1841.853283] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1841.864937] Lustre: Skipped 26 previous similar messages [ 1841.932317] Lustre: 66938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1848.810486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1848.816728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.824350] Lustre: Skipped 1 previous similar message [ 1848.828472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1853.412912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1854.769417] Lustre: server umount lustre-OST0000 complete [ 1859.043167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.052104] Lustre: Skipped 1 previous similar message [ 1864.831564] Lustre: server umount lustre-MDT0000 complete [ 1868.443283] LustreError: 64366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298805 with bad export cookie 1089337492319138681 [ 1868.452402] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1868.464113] LustreError: 64366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1868.750204] Lustre: server umount lustre-MDT0001 complete [ 1878.382415] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1880.797218] Key type lgssc unregistered [ 1881.053721] LNet: 68144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1881.060398] LNetError: 68144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1881.088129] LNet: Removed LNI 192.168.202.137@tcp [ 1881.797919] Key type .llcrypt unregistered [ 1881.799819] Key type ._llcrypt unregistered [ 1906.092174] Key type ._llcrypt registered [ 1906.095126] Key type .llcrypt registered [ 1906.269890] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1907.603525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1907.828499] alg: No test for adler32 (adler32-zlib) [ 1908.995935] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 1909.366100] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1911.128267] Key type lgssc registered [ 1912.225646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1920.996639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1921.015522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.622900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.822591] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1932.776512] LustreError: 69555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1932.805738] LustreError: 69555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1937.903431] LustreError: 69556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1938.316820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.831545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1943.930906] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1952.813910] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1959.569433] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1966.978118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1967.570206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.524216] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1983.873827] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1993.150787] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2003.438847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.464752] Lustre: Skipped 1 previous similar message [ 2003.466905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2003.487921] Lustre: Skipped 2 previous similar messages [ 2007.866519] Lustre: server umount lustre-MDT0000 complete [ 2008.553106] LustreError: 69550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.579067] LustreError: 69550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2012.126911] LustreError: 69535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298949 with bad export cookie 12953359918149040145 [ 2012.135269] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2012.139580] LustreError: 69535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2012.428476] Lustre: server umount lustre-MDT0001 complete [ 2026.832436] Lustre: server umount lustre-OST0000 complete [ 2052.341204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2052.607326] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2052.644511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2053.065902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.973798] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2067.106646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.189280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2067.248392] Lustre: 74429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2067.772891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2071.768908] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2079.811818] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2085.567600] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2093.644282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2093.809172] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2094.467894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.790995] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2111.869932] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2113.483373] Lustre: 77004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2115.679828] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2123.233417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2123.251555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.285275] Lustre: Skipped 1 previous similar message [ 2123.294460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2126.454285] Lustre: server umount lustre-OST0000 complete [ 2133.987112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.987551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2133.995692] Lustre: Skipped 1 previous similar message [ 2133.996845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.013950] Lustre: Skipped 2 previous similar messages [ 2137.585902] Lustre: server umount lustre-MDT0000 complete [ 2139.106981] LustreError: 74436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.138224] LustreError: 74436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2141.467824] LustreError: 74420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299078 with bad export cookie 12953359918149040978 [ 2141.471235] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2141.482867] LustreError: 74420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2141.773960] Lustre: server umount lustre-MDT0001 complete [ 2152.329448] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2155.188611] Key type lgssc unregistered [ 2155.567597] LNet: 78211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.573363] LNetError: 78211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.584905] LNet: Removed LNI 192.168.202.137@tcp [ 2156.293575] Key type .llcrypt unregistered [ 2156.296197] Key type ._llcrypt unregistered [ 2184.370408] Key type ._llcrypt registered [ 2184.372769] Key type .llcrypt registered [ 2184.467388] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2185.345591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2185.368934] alg: No test for adler32 (adler32-zlib) [ 2186.578078] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 2186.821098] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2188.506074] Key type lgssc registered [ 2189.750448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2197.472880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2197.489285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.888290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.696526] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2209.250258] LustreError: 79621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.288707] LustreError: 79621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2213.213789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2213.657778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2218.283750] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2226.091948] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2231.604499] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2240.506735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2241.426843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.848512] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2260.344143] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2266.588500] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2267.538838] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2269.396813] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2269.400656] Lustre: Skipped 1 previous similar message [ 2272.501222] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2272.509407] Lustre: Skipped 2 previous similar messages [ 2276.572781] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2276.576251] Lustre: Skipped 3 previous similar messages [ 2285.658931] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2285.667068] Lustre: Skipped 7 previous similar messages [ 2301.989955] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2301.993291] Lustre: Skipped 15 previous similar messages [ 2334.430533] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2334.441228] Lustre: Skipped 32 previous similar messages [ 2377.467364] Lustre: 87728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2380.739851] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 07:35:15 (1789299315) [ 2382.784754] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2384.791859] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 07:35:19 (1789299319) [ 2391.011812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2391.014578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2391.031927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2391.048029] Lustre: Skipped 1 previous similar message [ 2391.076749] Lustre: Skipped 1 previous similar message [ 2395.662880] Lustre: server umount lustre-OST0000 complete [ 2401.260960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2401.264848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2401.288152] Lustre: Skipped 1 previous similar message [ 2406.372907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2406.393322] Lustre: Skipped 2 previous similar messages [ 2406.617607] Lustre: server umount lustre-MDT0000 complete [ 2410.705881] LustreError: 79602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299347 with bad export cookie 2048575571355982546 [ 2410.722522] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2411.092414] Lustre: server umount lustre-MDT0001 complete [ 2423.496642] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2426.422986] Key type lgssc unregistered [ 2426.623650] LNet: 88985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2426.630334] LNetError: 88985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.648218] LNet: Removed LNI 192.168.202.137@tcp [ 2427.449400] Key type .llcrypt unregistered [ 2427.451664] Key type ._llcrypt unregistered [ 2432.632661] Key type ._llcrypt registered [ 2432.638179] Key type .llcrypt registered [ 2432.823519] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2438.834344] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules [ 2440.378994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2440.709265] alg: No test for adler32 (adler32-zlib) [ 2441.961450] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 2442.280475] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2444.000383] Key type lgssc registered [ 2445.622868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2447.252873] LustreError: 90231:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2468.562898] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2470.832584] LustreError: 90231:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2479.061236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2479.085545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2480.540868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2485.441077] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2489.836143] LustreError: 91261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.852012] LustreError: 91261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2494.945308] LustreError: 91260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2494.991403] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 07:37:09 (1789299429) [ 2500.081461] LustreError: 91261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2500.910350] LustreError: 91292:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2500.926262] LustreError: 91292:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2500.943280] LustreError: 91292:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2501.478937] Lustre: server umount lustre-MDT0000 complete [ 2517.895755] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2520.783201] Key type lgssc unregistered [ 2521.075266] LNet: 92786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2521.080424] LNetError: 92786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2521.098559] LNet: Removed LNI 192.168.202.137@tcp [ 2521.991038] Key type .llcrypt unregistered [ 2521.994903] Key type ._llcrypt unregistered [ 2541.071939] Key type ._llcrypt registered [ 2541.074257] Key type .llcrypt registered [ 2541.181841] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2542.436511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2542.564927] alg: No test for adler32 (adler32-zlib) [ 2543.829735] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 2544.154852] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2545.928263] Key type lgssc registered [ 2547.358342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2554.570692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2554.583415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2556.108676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.908080] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2566.629490] LustreError: 94014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2566.640908] LustreError: 94014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2569.083497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.840208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2574.635959] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2581.521430] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2587.540615] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2594.545675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2594.726169] LustreError: 95840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.141613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2601.681221] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2609.229141] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2661.536730] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 07:39:55 (1789299595) [ 2663.425829] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2665.844670] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 07:40:00 (1789299600) [ 2672.103554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2672.111310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.130959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2673.122524] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2673.127947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2673.163381] Lustre: Skipped 2 previous similar messages [ 2677.235563] Lustre: server umount lustre-MDT0000 complete [ 2678.245529] LustreError: 94009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.268878] LustreError: 94009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2681.556204] LustreError: 97192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299618 with bad export cookie 15237888374883372617 [ 2681.557917] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2681.566298] LustreError: 97192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2681.897944] Lustre: server umount lustre-MDT0001 complete [ 2696.722563] Lustre: server umount lustre-OST0000 complete [ 2710.113278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2710.537815] LustreError: 98309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.774551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.783114] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2716.132606] LustreError: 98310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.168821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2724.946438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2729.923287] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2738.766625] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2744.759340] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2760.889907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2761.148736] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2761.154717] LustreError: 98299:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2761.179627] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2761.186787] LustreError: 100393:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2761.190140] LustreError: 100393:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2761.193113] LustreError: 100393:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2761.248482] Lustre: server umount lustre-OST0000 complete [ 2761.251142] LustreError: 100393:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2771.625811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2771.815959] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2771.822410] LustreError: 98299:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2771.860787] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2771.874874] LustreError: 100862:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2771.880189] LustreError: 100862:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2771.883290] LustreError: 100862:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2771.964033] Lustre: server umount lustre-OST0000 complete [ 2771.967702] LustreError: 100862:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2783.038770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2783.239567] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2783.248279] Lustre: Found index 0 for lustre-OST0000, updating log [ 2783.259446] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2783.335556] Lustre: lustre-OST0000: new disk, initializing [ 2783.338828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2783.343975] Lustre: 101396:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2783.582183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2790.963239] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2795.636961] Lustre: Failing over lustre-OST0000 [ 2795.950665] Lustre: server umount lustre-OST0000 complete [ 2799.669680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.675130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2799.676961] Lustre: Skipped 2 previous similar messages [ 2799.688463] Lustre: Skipped 1 previous similar message [ 2804.708777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2804.717355] Lustre: Skipped 1 previous similar message [ 2806.081708] Lustre: server umount lustre-MDT0000 complete [ 2809.831772] LustreError: 98310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.851958] LustreError: 98310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2810.861899] LustreError: 98291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299748 with bad export cookie 15237888374883375361 [ 2810.864736] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2810.874828] LustreError: 98291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2821.322445] Lustre: server umount lustre-MDT0001 complete [ 2831.195979] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 07:42:46 (1789299766) [ 2851.346284] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2862.033272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2862.493108] LustreError: 103744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.703460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2868.220608] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2876.780519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.093740] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2885.203789] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2893.484803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2894.543544] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2902.682575] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2902.700649] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2902.727710] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2902.728313] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2912.968577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2913.074765] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2913.139293] Lustre: lustre-OST0001: new disk, initializing [ 2913.142618] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2913.147960] Lustre: 106063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2913.424325] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2913.429297] Lustre: Skipped 2 previous similar messages [ 2921.658515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2921.674252] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2921.873129] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2922.289442] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2931.845912] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2941.886321] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2947.555741] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.562636] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2947.579860] Lustre: Skipped 2 previous similar messages [ 2947.592047] Lustre: Skipped 2 previous similar messages [ 2953.646468] Lustre: server umount lustre-MDT0001 complete [ 2956.089970] LustreError: 103739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.126151] LustreError: 103739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2965.371305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.433794] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2965.436421] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2965.474087] Lustre: 103731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2967.285406] LustreError: 107617:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2967.293926] LustreError: 107617:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2967.307765] LustreError: 103736:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2967.442306] LustreError: 107635:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.137@tcp: cfg command failed: rc = -17 [ 2967.458330] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2967.458330] [ 2970.913454] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2971.646480] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2981.265184] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 07:45:16 (1789299916) [ 2999.010417] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3016.918381] Lustre: 109676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3040.682804] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3044.524226] Lustre: 110811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3130.336698] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.351182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.354922] Lustre: Skipped 4 previous similar messages [ 3136.641843] Lustre: server umount lustre-MDT0000 complete [ 3140.406239] LustreError: 103726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789300077 with bad export cookie 15237888374883376362 [ 3140.412131] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3140.420169] LustreError: 103726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3140.591976] LustreError: 111170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.595367] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.613654] LustreError: 111170:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3140.639236] Lustre: Skipped 3 previous similar messages [ 3140.815704] Lustre: server umount lustre-MDT0001 complete [ 3147.148623] Lustre: server umount lustre-OST0000 complete [ 3151.909293] Lustre: server umount lustre-OST0001 complete [ 3158.421395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3170.007604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3179.233628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3189.904574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3212.724994] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3222.322219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3222.335464] Lustre: lustre-MDT0000: reset Object Index mappings [ 3222.910250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3222.916841] Lustre: Skipped 1 previous similar message [ 3226.676406] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3234.762317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3234.775283] Lustre: lustre-MDT0001: reset Object Index mappings [ 3239.492077] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3242.239629] Lustre: 115911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3248.096681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3254.790956] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3259.963689] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3263.347751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3267.080330] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3267.084183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3267.084182] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3270.486314] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3278.344968] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3281.550653] Lustre: 117788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3330.988669] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 07:51:05 (1789300265) [ 3358.180411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3358.189290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3358.207958] Lustre: Skipped 1 previous similar message [ 3358.216626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3358.219369] Lustre: Skipped 6 previous similar messages [ 3363.858812] Lustre: server umount lustre-MDT0000 complete [ 3367.397930] LustreError: 117522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3367.416020] LustreError: 117522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3367.687595] LustreError: 115913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789300304 with bad export cookie 15237888374883423234 [ 3367.695899] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3367.702651] LustreError: 115913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3374.396167] Lustre: server umount lustre-MDT0001 complete [ 3384.785256] Lustre: server umount lustre-OST0000 complete [ 3403.232186] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3403.417638] Lustre: server umount lustre-OST0001 complete [ 3410.765409] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 3422.421145] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3458.813363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3459.101989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3459.144941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3459.240175] Lustre: lustre-MDT0000: new disk, initializing [ 3459.292037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3459.304316] Lustre: Skipped 3 previous similar messages [ 3459.315908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3463.857364] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3470.688283] Lustre: server umount lustre-MDT0000 complete [ 3481.554079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3481.985387] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3482.001475] Lustre: 122063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3482.013704] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3487.504420] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3492.551509] Lustre: server umount lustre-MDT0000 complete [ 3504.741881] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 07:53:59 (1789300439) [ 3506.858133] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3508.876629] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 07:54:03 (1789300443) [ 3527.762545] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3538.439728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3543.669403] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3552.746396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3552.882164] Lustre: 124112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3552.927862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3552.931416] Lustre: Skipped 1 previous similar message [ 3553.009089] Lustre: lustre-MDT0001: new disk, initializing [ 3553.196781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3553.231645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3558.084170] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3563.289873] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3570.953645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3571.380618] Lustre: lustre-OST0000: new disk, initializing [ 3571.383938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3571.391106] Lustre: 125690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3573.004545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3573.036830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3573.279581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3580.407685] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3592.930741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3593.171510] Lustre: lustre-OST0001: new disk, initializing [ 3593.175236] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3593.194286] Lustre: 126563:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3594.900977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3594.930640] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3595.119610] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3600.601298] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3611.673280] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3615.696881] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3982.281843] Lustre: 132985:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3982.292320] Lustre: 132985:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3982.298533] Lustre: 132985:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3982.317764] Lustre: 132985:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3982.324713] Lustre: 132985:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3982.334450] Lustre: 132985:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4260.602583] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:06:35 (1789301195) [ 4275.170967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4275.183400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4275.203337] Lustre: Skipped 5 previous similar messages [ 4275.212518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4275.230937] Lustre: Skipped 6 previous similar messages [ 4277.324368] Lustre: server umount lustre-MDT0000 complete [ 4281.219355] LustreError: 133005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789301218 with bad export cookie 15237888374883490287 [ 4281.222645] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4281.234593] LustreError: 133005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4281.314127] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.315465] LustreError: 124121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4281.323602] Lustre: Skipped 3 previous similar messages [ 4281.327578] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4281.340820] LustreError: 124121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4281.363829] Lustre: Skipped 5 previous similar messages [ 4287.834499] Lustre: server umount lustre-MDT0001 complete [ 4305.889154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4306.129647] Lustre: server umount lustre-OST0000 complete [ 4324.320599] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4324.525385] Lustre: server umount lustre-OST0001 complete [ 4331.762655] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 4343.569235] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4380.348885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4380.560581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4380.592685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4380.667747] Lustre: lustre-MDT0000: new disk, initializing [ 4380.803122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4380.812185] Lustre: Skipped 5 previous similar messages [ 4380.834205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4385.133379] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4395.499135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4395.697995] Lustre: 136036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4395.755882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4395.763692] Lustre: Skipped 1 previous similar message [ 4395.925377] Lustre: lustre-MDT0001: new disk, initializing [ 4396.121275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4396.135268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4401.854345] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4412.173323] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4419.828431] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4427.677743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4428.054485] Lustre: lustre-OST0000: new disk, initializing [ 4428.062452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4428.076249] Lustre: 138008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4428.241113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.251368] Lustre: Skipped 1 previous similar message [ 4429.536661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4429.548174] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4429.790153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4435.790598] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4446.635975] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4457.141564] Lustre: lustre-MDD0000: changelog on [ 4460.133095] Lustre: lustre-MDD0001: changelog on [ 6440.839967] Lustre: lustre-MDD0001: changelog off [ 6443.025302] Lustre: 139451:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6444.074628] Lustre: lustre-MDD0000: changelog off [ 6450.165474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6450.178710] Lustre: Skipped 1 previous similar message [ 6450.187795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6450.192571] Lustre: Skipped 3 previous similar messages [ 6455.278090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6455.285177] Lustre: Skipped 1 previous similar message [ 6456.338522] Lustre: server umount lustre-OST0000 complete [ 6460.402145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6460.410589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6460.419653] Lustre: Skipped 2 previous similar messages [ 6460.432787] Lustre: Skipped 1 previous similar message [ 6465.508406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6465.514021] Lustre: Skipped 1 previous similar message [ 6474.208139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6474.427485] Lustre: server umount lustre-MDT0000 complete [ 6475.753732] LustreError: 136893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6475.784343] LustreError: 136893:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6478.379847] LustreError: 136029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303415 with bad export cookie 15237888374884306193 [ 6478.387413] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6478.399168] LustreError: 136029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6478.706164] Lustre: server umount lustre-MDT0001 complete [ 6490.297671] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6493.308426] Key type lgssc unregistered [ 6493.653127] LNet: 147148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6493.661814] LNetError: 147148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6493.682869] LNet: Removed LNI 192.168.202.137@tcp [ 6494.569277] Key type .llcrypt unregistered [ 6494.572328] Key type ._llcrypt unregistered [ 6515.004817] Key type ._llcrypt registered [ 6515.009987] Key type .llcrypt registered [ 6515.131343] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 6531.826573] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6533.363783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6533.390471] alg: No test for adler32 (adler32-zlib) [ 6534.515495] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 6534.951041] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6536.776761] Key type lgssc registered [ 6537.979308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6583.497978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6583.525637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6585.008306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6585.065571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6585.193084] Lustre: lustre-MDT0000: new disk, initializing [ 6585.315070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6585.359181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6590.344163] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6601.793122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6601.940703] Lustre: 150353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6602.015620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6602.018771] Lustre: Skipped 1 previous similar message [ 6602.076910] Lustre: lustre-MDT0001: new disk, initializing [ 6602.161057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6602.193562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6602.206675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6607.281392] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6618.602803] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6626.482661] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6633.899372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6634.278503] Lustre: lustre-OST0000: new disk, initializing [ 6634.282947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6634.292452] Lustre: 152321:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6634.429924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6641.728114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6641.744520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6641.882740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6642.084789] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6652.491549] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6659.691618] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_module obdecho/obdecho [ 6662.286681] LustreError: 153326:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6662.305230] LustreError: 153326:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6664.692271] LustreError: 153422:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6683.110514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6683.119666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6683.147947] Lustre: Skipped 2 previous similar messages [ 6683.166037] Lustre: Skipped 2 previous similar messages [ 6686.955342] Lustre: server umount lustre-MDT0000 complete [ 6692.288511] LustreError: 152331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303629 with bad export cookie 11799599150136233191 [ 6692.301719] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6692.318413] LustreError: 152331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6692.808931] Lustre: server umount lustre-MDT0001 complete [ 6708.600827] Lustre: server umount lustre-OST0000 complete [ 6732.391208] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 6744.693323] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6785.936358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6786.238171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6786.266064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6786.323416] Lustre: lustre-MDT0000: new disk, initializing [ 6786.385038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6786.395738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6791.289244] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6802.838727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6802.951759] Lustre: 157142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6803.005633] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6803.009840] Lustre: Skipped 1 previous similar message [ 6803.078941] Lustre: lustre-MDT0001: new disk, initializing [ 6803.187972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6803.233660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6803.247836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6807.721309] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6817.899487] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6823.625749] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6830.691541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6830.963771] Lustre: lustre-OST0000: new disk, initializing [ 6830.967444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6830.976583] Lustre: 159108:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6831.058413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6837.153753] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6838.879382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6838.891727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6838.992362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6846.643460] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6852.023805] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6852.326509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6858.953714] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6859.191127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6864.881299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6864.886363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6864.896084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6864.903902] Lustre: Skipped 1 previous similar message [ 6864.922094] Lustre: Skipped 1 previous similar message [ 6868.219583] Lustre: server umount lustre-OST0000 complete [ 6875.113913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6875.121493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6875.131989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6875.160130] LustreError: Skipped 1 previous similar message [ 6878.966874] Lustre: server umount lustre-MDT0000 complete [ 6880.230853] LustreError: 159117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6880.265281] LustreError: 159117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6882.841297] LustreError: 159107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303820 with bad export cookie 11799599150136234143 [ 6882.846947] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6882.852475] LustreError: 159107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6883.090210] Lustre: server umount lustre-MDT0001 complete [ 6887.569871] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 08:50:22 (1789303822) [ 6889.966401] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6892.549086] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 08:50:26 (1789303826) [ 6902.015232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6902.579046] LustreError: 161522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6902.674149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6907.430812] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6907.881886] LustreError: 161523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6911.972238] LustreError: 161522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6916.900878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6917.090250] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6917.092513] Lustre: Skipped 1 previous similar message [ 6921.765857] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6929.718860] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6935.607056] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6941.494792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6941.760741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6941.772574] Lustre: Skipped 1 previous similar message [ 6948.067604] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6957.242559] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6967.265826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6967.271241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6967.294449] Lustre: Skipped 3 previous similar messages [ 6967.322896] Lustre: Skipped 2 previous similar messages [ 6970.430213] Lustre: server umount lustre-MDT0000 complete [ 6972.395369] LustreError: 162262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6972.429591] LustreError: 162262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6975.209873] LustreError: 161503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303912 with bad export cookie 11799599150136235025 [ 6975.218302] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6975.221289] LustreError: 161503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6977.510762] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6977.539230] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6977.554095] Lustre: Skipped 1 previous similar message [ 6980.513360] LustreError: 162913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6980.546259] LustreError: 162913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6981.647594] Lustre: server umount lustre-MDT0001 complete [ 6988.205868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7001.767533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7001.929303] LustreError: 165364:0:(llog.c:1655:llog_backup()) MGC192.168.202.137@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7001.936885] Lustre: 165364:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.137@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7005.157555] LustreError: 165369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7039.790737] LustreError: 165369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7039.838420] LustreError: 165369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 7062.496646] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7062.506074] Lustre: 165364:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7103.505666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7105.329065] LustreError: 165370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7105.365676] LustreError: 165370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 7108.600981] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7109.287451] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7119.782716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.087623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7151.108078] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7152.814675] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7162.164666] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7163.677940] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7170.061965] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7172.145979] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7195.653429] Lustre: server umount lustre-MDT0000 complete [ 7197.166820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7197.172539] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7197.189862] Lustre: Skipped 2 previous similar messages [ 7200.541357] LustreError: 164305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304137 with bad export cookie 11799599150136241521 [ 7200.544717] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7200.568601] LustreError: 164305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7200.916268] Lustre: server umount lustre-MDT0001 complete [ 7216.112081] Lustre: server umount lustre-OST0000 complete [ 7240.009107] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 7250.962656] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7293.892834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7294.209951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7294.263715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7294.372817] Lustre: lustre-MDT0000: new disk, initializing [ 7294.465531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7294.470661] Lustre: Skipped 1 previous similar message [ 7294.488067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7299.831491] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7310.543886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7310.696940] Lustre: 170475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7311.151689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7316.636369] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7326.543424] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7332.801573] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7340.895210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7341.138548] Lustre: lustre-OST0000: new disk, initializing [ 7341.141589] Lustre: Skipped 1 previous similar message [ 7341.147556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7341.151151] Lustre: Skipped 2 previous similar messages [ 7341.155816] Lustre: 172444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7348.169528] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7348.299990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7348.308784] Lustre: Skipped 1 previous similar message [ 7348.311377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7348.438700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7357.758034] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7362.739951] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7362.966711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7368.228582] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7368.493945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7374.313535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7374.319766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7374.321780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7374.321787] Lustre: Skipped 2 previous similar messages [ 7374.376252] Lustre: Skipped 1 previous similar message [ 7377.279253] Lustre: server umount lustre-OST0000 complete [ 7383.011751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7383.029679] LustreError: Skipped 1 previous similar message [ 7388.677716] Lustre: server umount lustre-MDT0000 complete [ 7389.672570] LustreError: 170482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7389.720375] LustreError: 170482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 7393.069811] LustreError: 170466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304330 with bad export cookie 11799599150136258321 [ 7393.073786] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7393.086748] LustreError: 170466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7393.697866] Lustre: server umount lustre-MDT0001 complete [ 7406.387068] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:59:00 (1789304340) [ 7414.259033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7419.739553] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7428.420890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7433.340553] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7441.974979] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7448.395746] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7456.364063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7462.981857] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7470.194856] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7484.945754] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 09:00:19 (1789304419) [ 7490.533936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7490.553101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7490.569296] Lustre: Skipped 2 previous similar messages [ 7490.581145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7490.589897] Lustre: Skipped 4 previous similar messages [ 7494.992944] Lustre: server umount lustre-OST0000 complete [ 7505.933908] Lustre: server umount lustre-MDT0000 complete [ 7510.415626] LustreError: 174836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304447 with bad export cookie 11799599150136259203 [ 7510.422548] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7510.424827] LustreError: 174836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7510.795341] Lustre: server umount lustre-MDT0001 complete [ 7522.731308] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 7525.903638] Key type lgssc unregistered [ 7526.222561] LNet: 178595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7526.227835] LNetError: 178595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7526.256915] LNet: Removed LNI 192.168.202.137@tcp [ 7527.220954] Key type .llcrypt unregistered [ 7527.224803] Key type ._llcrypt unregistered [ 7546.059142] Key type ._llcrypt registered [ 7546.061670] Key type .llcrypt registered [ 7546.185659] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7547.471343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7547.491955] alg: No test for adler32 (adler32-zlib) [ 7548.552475] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 7548.815405] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 7550.488185] Key type lgssc registered [ 7551.579281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7558.757502] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7558.780500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7574.304441] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7579.424466] LustreError: 179835:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [ 7584.544506] LustreError: 179835:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7584.551229] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7584.556069] LustreError: MGC192.168.202.137@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7584.562860] LustreError: 179835:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7584.569856] LustreError: 179835:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7584.575604] LustreError: 179835:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7584.649867] Lustre: server umount lustre-OST0000 complete [ 7584.653612] LustreError: 179835:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7593.600704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7595.072894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7600.001653] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7605.223942] LustreError: 180334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7605.251176] LustreError: 180334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7609.331496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7609.745450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7614.718754] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7622.954280] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7629.212765] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7636.651394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7636.847348] LustreError: 182152:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7636.864729] Lustre: 182152:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.137@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7637.073838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7643.396929] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7650.805721] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7653.923365] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7653.929300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7653.939343] Lustre: Skipped 2 previous similar messages [ 7653.958262] Lustre: Skipped 2 previous similar messages [ 7658.980661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7658.984843] Lustre: Skipped 1 previous similar message [ 7664.099279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7664.113877] Lustre: Skipped 2 previous similar messages [ 7667.168125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7667.428852] Lustre: server umount lustre-MDT0000 complete [ 7669.229062] LustreError: 180334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7669.254696] LustreError: 180334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7670.984845] LustreError: 180315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304608 with bad export cookie 16233910906018903524 [ 7670.993720] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7671.001539] LustreError: 180315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7672.416290] LustreError: 181063:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000038a97b31 x1876221739694080/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7672.653574] Lustre: server umount lustre-MDT0001 complete [ 7687.318506] Lustre: server umount lustre-OST0000 complete [ 7697.104620] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 7707.404269] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7746.115926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7746.294820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7746.330230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7746.422830] Lustre: lustre-MDT0000: new disk, initializing [ 7746.548465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7746.566762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7751.481826] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7762.144867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7762.330204] Lustre: 185677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7762.413881] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7762.429244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7762.439405] Lustre: Skipped 1 previous similar message [ 7762.512898] Lustre: lustre-MDT0001: new disk, initializing [ 7762.596909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7762.635953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7762.658981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7767.615782] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7778.369851] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7784.773201] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7792.694462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7792.910819] Lustre: lustre-OST0000: new disk, initializing [ 7792.913743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7792.922175] Lustre: 187642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7793.010927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7798.358109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7798.374818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7798.466513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7799.493979] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7809.774293] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7814.262361] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7814.445417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7819.454484] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7819.630431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7824.354070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7824.356518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7824.369504] LustreError: Skipped 1 previous similar message [ 7824.400424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7824.422730] Lustre: Skipped 1 previous similar message [ 7827.539488] Lustre: server umount lustre-OST0000 complete [ 7834.593943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7834.607621] Lustre: Skipped 1 previous similar message [ 7834.630329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7834.637856] Lustre: Skipped 1 previous similar message [ 7837.518348] Lustre: server umount lustre-MDT0000 complete [ 7839.715369] LustreError: 187072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7839.732648] LustreError: 187072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7841.663912] LustreError: 185667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304778 with bad export cookie 16233910906018904238 [ 7841.668095] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7841.673810] LustreError: 185667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7842.167964] Lustre: server umount lustre-MDT0001 complete [ 7852.747948] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 09:06:27 (1789304787) [ 7872.135741] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 7875.402560] Key type lgssc unregistered [ 7875.753843] LNet: 190593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7875.760956] LNetError: 190593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7875.782963] LNet: Removed LNI 192.168.202.137@tcp [ 7876.748674] Key type .llcrypt unregistered [ 7876.753551] Key type ._llcrypt unregistered [ 7898.087450] Key type ._llcrypt registered [ 7898.090703] Key type .llcrypt registered [ 7898.193944] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7899.403661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7899.486725] alg: No test for adler32 (adler32-zlib) [ 7900.647617] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 7901.015838] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 7902.768704] Key type lgssc registered [ 7904.030975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7912.616420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7912.632044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7914.256635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7919.051506] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7924.706610] LustreError: 191805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7924.718842] LustreError: 191805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7927.829688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7928.257226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7933.704514] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7942.182937] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7948.690305] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7957.817670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7957.990833] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7957.993349] LustreError: 193669:0:(llog.c:1696:llog_backup()) MGC192.168.202.137@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7957.999115] Lustre: 193669:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.137@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7958.100416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7964.944899] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7973.021931] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7979.492815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7979.504295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7979.529902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7980.005301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7980.025046] Lustre: Skipped 1 previous similar message [ 7980.029682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7980.032226] Lustre: Skipped 1 previous similar message [ 7981.710580] Lustre: server umount lustre-MDT0000 complete [ 7985.017903] LustreError: 191788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304922 with bad export cookie 12448693695623261132 [ 7985.025358] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7985.121987] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7985.129539] LustreError: 191800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7985.131379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7985.250662] Lustre: server umount lustre-MDT0001 complete [ 7991.457513] Lustre: server umount lustre-OST0000 complete [ 8001.875887] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 8012.857550] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8049.386716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8049.611162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8049.636625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8049.713710] Lustre: lustre-MDT0000: new disk, initializing [ 8049.783514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8049.805759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8054.895571] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8065.259078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8065.368635] Lustre: 197193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8065.398261] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8065.400792] Lustre: Skipped 1 previous similar message [ 8065.404729] LustreError: 197973:0:(llog.c:1696:llog_backup()) MGC192.168.202.137@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8065.410780] LustreError: 197973:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8065.415596] Lustre: 197973:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.137@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8065.433194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8065.440085] Lustre: Skipped 1 previous similar message [ 8065.509980] Lustre: lustre-MDT0001: new disk, initializing [ 8065.597980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8065.636596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8065.667909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8070.378205] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8079.882168] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8086.817806] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8094.263140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8094.670823] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8094.675152] Lustre: Skipped 1 previous similar message [ 8094.681319] LustreError: 199134:0:(llog.c:1696:llog_backup()) MGC192.168.202.137@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8094.701592] LustreError: 199134:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8094.713899] Lustre: 199134:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.137@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8094.723215] Lustre: 199134:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8094.756736] Lustre: lustre-OST0000: new disk, initializing [ 8094.771079] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8094.791781] Lustre: 199158:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8094.907475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8101.470917] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8102.987864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8103.007524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8103.131801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8111.361811] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8116.154488] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8116.399317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8121.145705] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8121.379217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8123.876102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8123.878903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8123.895020] Lustre: Skipped 1 previous similar message [ 8123.895660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8123.909491] Lustre: Skipped 1 previous similar message [ 8129.001490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8129.004026] Lustre: Skipped 1 previous similar message [ 8129.376699] Lustre: server umount lustre-OST0000 complete [ 8134.117707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8134.128112] Lustre: Skipped 1 previous similar message [ 8139.234841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8139.245629] Lustre: Skipped 3 previous similar messages [ 8139.887520] Lustre: server umount lustre-MDT0000 complete [ 8143.195191] LustreError: 197185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305080 with bad export cookie 12448693695623261853 [ 8143.203868] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8143.207940] LustreError: 197185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8143.528232] Lustre: server umount lustre-MDT0001 complete [ 8152.564707] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 09:11:27 (1789305087) [ 8170.009975] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8179.989857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8180.265223] LustreError: 202287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8180.310297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8183.735906] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8185.316560] LustreError: 202288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8190.107615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8194.693300] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8197.262531] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8203.747760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8210.295700] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8219.233537] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8219.422768] Lustre: lustre-OST0001: new disk, initializing [ 8219.426622] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8219.431499] Lustre: 204594:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8219.537310] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8219.544311] Lustre: Skipped 2 previous similar messages [ 8225.339387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8225.347895] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8225.451418] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8226.422409] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8236.920403] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8246.102692] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8247.888699] Lustre: DEBUG MARKER: ADD OST3 [ 8252.655295] loop: module loaded [ 8252.706597] loop0: detected capacity change from 0 to 204800000 [ 8252.991052] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8253.009773] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8253.031757] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8258.993941] loop0: detected capacity change from 0 to 204800000 [ 8259.949260] Lustre: DEBUG MARKER: STOP OST3 [ 8263.538916] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8263.701247] Lustre: lustre-OST0003: new disk, initializing [ 8263.704720] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8263.710441] Lustre: 206581:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8265.074518] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8265.186137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8265.204687] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8265.243087] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8267.109581] LustreError: 206765:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8267.133794] Lustre: Failing over lustre-OST0003 [ 8267.310354] Lustre: server umount lustre-OST0003 complete [ 8269.301808] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8270.314771] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8270.325766] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8270.328233] LustreError: 203774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8270.359705] Lustre: Skipped 1 previous similar message [ 8276.846080] Lustre: DEBUG MARKER: START OST3 again [ 8280.567764] LustreError: 203774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8280.583793] LustreError: 203774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8282.404032] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8282.682408] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8284.587660] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8284.645856] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8284.647148] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8288.536065] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8308.020117] Lustre: Failing over lustre-OST0003 [ 8308.108500] Lustre: server umount lustre-OST0003 complete [ 8308.201686] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8308.220611] LustreError: 203774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8308.241714] LustreError: 203774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8328.675224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8328.683655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8328.709514] Lustre: Skipped 3 previous similar messages [ 8331.790844] Lustre: server umount lustre-MDT0000 complete [ 8335.036690] LustreError: 205363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305272 with bad export cookie 12448693695623262735 [ 8335.046640] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8335.048582] LustreError: 205363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8340.965332] LustreError: 203914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8340.990937] LustreError: 203914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 8341.451575] Lustre: server umount lustre-MDT0001 complete [ 8344.592928] Lustre: server umount lustre-OST0000 complete [ 8347.785379] Lustre: server umount lustre-OST0001 complete [ 8354.318675] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 8362.086191] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8394.975123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8395.237698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8395.272382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8395.403273] Lustre: lustre-MDT0000: new disk, initializing [ 8395.486728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8395.493101] Lustre: Skipped 2 previous similar messages [ 8395.502379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8399.799276] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8409.160206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8409.303063] Lustre: 212047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8409.572573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8414.188200] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8422.712571] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8428.116233] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8433.909775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8434.103390] Lustre: lustre-OST0000: new disk, initializing [ 8434.107424] Lustre: Skipped 1 previous similar message [ 8434.112595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8434.119230] Lustre: Skipped 2 previous similar messages [ 8434.123729] Lustre: 214014:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8435.403591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8435.411376] Lustre: Skipped 1 previous similar message [ 8435.415813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8435.494512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8440.069777] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8448.195262] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8452.265666] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8452.422639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8456.247692] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8456.423645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8461.288055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8461.290495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8461.297163] LustreError: Skipped 1 previous similar message [ 8461.297195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8461.297201] Lustre: Skipped 4 previous similar messages [ 8461.317638] Lustre: Skipped 8 previous similar messages [ 8472.032136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8472.211517] Lustre: server umount lustre-OST0000 complete [ 8476.132172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8481.765156] LustreError: 212054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8481.775176] LustreError: 212054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8481.809745] Lustre: server umount lustre-MDT0000 complete [ 8485.448621] LustreError: 212038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305422 with bad export cookie 12448693695623265031 [ 8485.455830] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8485.462443] LustreError: 212038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8485.756572] Lustre: server umount lustre-MDT0001 complete [ 8489.298881] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 09:17:04 (1789305424) [ 8503.962584] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 8513.729538] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8544.386070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8544.660585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8544.700442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8544.801546] Lustre: lustre-MDT0000: new disk, initializing [ 8544.902900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8544.905188] Lustre: Skipped 2 previous similar messages [ 8544.913757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8549.153340] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8558.743141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8558.900856] Lustre: 218622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8559.221039] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8563.330639] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8572.684526] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8578.300654] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8585.225798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8585.643662] Lustre: 220592:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8587.086247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8591.961425] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8600.649539] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8605.129597] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8605.337664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8609.620679] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8609.806093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8612.842641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8612.851304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8612.853302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8612.853310] Lustre: Skipped 7 previous similar messages [ 8612.888149] Lustre: Skipped 3 previous similar messages [ 8617.584922] Lustre: server umount lustre-OST0000 complete [ 8628.196466] LustreError: 218636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8628.211988] LustreError: 218636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8628.952260] Lustre: server umount lustre-MDT0000 complete [ 8631.986056] LustreError: 218615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305569 with bad export cookie 12448693695623265913 [ 8631.990091] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8631.996234] LustreError: 218615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8632.247653] Lustre: server umount lustre-MDT0001 complete [ 8639.204798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8643.608191] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8651.464836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8655.801772] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8663.054977] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8668.392401] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8674.383214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8680.287738] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8686.677763] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8697.314168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8697.320794] LustreError: Skipped 1 previous similar message [ 8702.124828] Lustre: server umount lustre-OST0000 complete [ 8711.705388] Lustre: server umount lustre-MDT0000 complete [ 8714.845919] LustreError: 222787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305652 with bad export cookie 12448693695623266795 [ 8714.852766] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8714.857674] LustreError: 222787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8715.152385] Lustre: server umount lustre-MDT0001 complete [ 8823.310871] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 8825.479061] Key type lgssc unregistered [ 8825.714961] LNet: 226471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8825.721755] LNetError: 226471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8825.733710] LNet: Removed LNI 192.168.202.137@tcp [ 8826.450165] Key type .llcrypt unregistered [ 8826.452866] Key type ._llcrypt unregistered [ 8850.780748] Key type ._llcrypt registered [ 8850.782936] Key type .llcrypt registered [ 8850.878814] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 8864.494260] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8865.486602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8865.549112] alg: No test for adler32 (adler32-zlib) [ 8866.617395] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 8866.791102] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 8868.480369] Key type lgssc registered [ 8869.370883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8896.957592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8896.992624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8898.174955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8898.211575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8898.271726] Lustre: lustre-MDT0000: new disk, initializing [ 8898.323393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8898.338293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8901.891250] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8909.751867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8909.857536] Lustre: 229814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8909.917140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8909.924162] Lustre: Skipped 1 previous similar message [ 8909.978797] Lustre: lustre-MDT0001: new disk, initializing [ 8910.047503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8910.077217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8910.087548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8913.367872] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8921.337599] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8926.895607] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8933.306433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8933.513223] Lustre: lustre-OST0000: new disk, initializing [ 8933.516461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8933.526167] Lustre: 231780:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8933.598788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8938.681615] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8942.664373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8942.675268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8942.748466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8946.826516] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8951.277238] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8951.433986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8956.009391] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8956.256604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8958.439568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8958.444570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8958.453914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8958.463664] LustreError: Skipped 1 previous similar message [ 8963.554198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8963.565193] Lustre: Skipped 2 previous similar messages [ 8968.680759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8968.686827] Lustre: Skipped 1 previous similar message [ 8972.256300] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8972.471737] Lustre: server umount lustre-OST0000 complete [ 8976.866696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8976.877406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8976.895906] Lustre: Skipped 1 previous similar message [ 8976.902900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8978.914155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8984.035206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8984.042324] Lustre: Skipped 3 previous similar messages [ 8990.176179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8990.381372] Lustre: server umount lustre-MDT0000 complete [ 8993.541807] LustreError: 229805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305930 with bad export cookie 12091750722631949653 [ 8993.543664] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8993.549839] LustreError: 229805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8993.930954] Lustre: server umount lustre-MDT0001 complete [ 9009.066465] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9016.839552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9017.158601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9021.140558] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9022.436878] LustreError: 234714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9022.448618] LustreError: 234714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9027.560779] LustreError: 234713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9028.250483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9028.531163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9032.113485] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9034.525390] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9039.168896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9039.384687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9043.707910] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9049.725682] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9049.885589] Lustre: lustre-OST0001: new disk, initializing [ 9049.890185] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9049.899859] Lustre: 237019:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9053.733580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9053.745117] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9053.826238] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9055.047620] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9064.468666] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9072.508731] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9082.033841] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 09:26:57 (1789306017) [ 9084.900717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9084.906855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9084.920853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9084.926466] Lustre: Skipped 2 previous similar messages [ 9089.643426] Lustre: server umount lustre-MDT0000 complete [ 9091.056072] LustreError: 238014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9091.066783] LustreError: 238014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9092.228926] LustreError: 234695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306029 with bad export cookie 12091750722631950535 [ 9092.235959] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9092.243937] LustreError: 234695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9092.505241] Lustre: server umount lustre-MDT0001 complete [ 9105.484414] Lustre: server umount lustre-OST0000 complete [ 9118.845563] Lustre: server umount lustre-OST0001 complete [ 9124.397332] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 9132.424769] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9160.126108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9160.327558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9160.358301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9160.428268] Lustre: lustre-MDT0000: new disk, initializing [ 9160.489420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9160.493211] Lustre: Skipped 1 previous similar message [ 9160.505197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9163.760337] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9171.797600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9171.883593] Lustre: 240963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9171.926619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9171.929508] Lustre: Skipped 1 previous similar message [ 9171.980883] Lustre: lustre-MDT0001: new disk, initializing [ 9172.083523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9172.099679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9174.989279] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9181.736343] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9185.658979] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9190.778963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9190.987841] Lustre: lustre-OST0000: new disk, initializing [ 9190.991673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9191.000525] Lustre: 242928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9192.292425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9192.306448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9192.382137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9195.525511] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9202.522720] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9205.989632] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9206.179942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9209.802381] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9209.936329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9212.899522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9212.903719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9212.906191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9212.906200] Lustre: Skipped 4 previous similar messages [ 9212.929412] Lustre: Skipped 4 previous similar messages [ 9217.503230] Lustre: server umount lustre-OST0000 complete [ 9223.137937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9223.139843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9226.284048] Lustre: server umount lustre-MDT0000 complete [ 9228.257316] LustreError: 240975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9228.266434] LustreError: 240975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9228.994839] LustreError: 240954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306166 with bad export cookie 12091750722631952663 [ 9228.997870] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9229.004573] LustreError: 240954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9229.339393] Lustre: server umount lustre-MDT0001 complete [ 9235.129528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9235.415791] LustreError: 245145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9235.457680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9235.461391] Lustre: Skipped 2 previous similar messages [ 9238.417691] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9244.071202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9247.156400] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9251.837255] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9255.527216] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9260.018448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9264.447674] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9269.231252] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9413.092285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9413.092985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9413.096837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9413.105358] Lustre: Skipped 2 previous similar messages [ 9413.124098] Lustre: Skipped 4 previous similar messages [ 9418.685692] Lustre: server umount lustre-OST0000 complete [ 9428.241456] Lustre: server umount lustre-MDT0000 complete [ 9428.456483] LustreError: 245146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9428.471027] LustreError: 245146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9431.750129] LustreError: 245126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306368 with bad export cookie 12091750722631953552 [ 9431.760665] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9431.761506] LustreError: 245126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9432.078800] Lustre: server umount lustre-MDT0001 complete [ 9534.463119] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 9536.654661] Key type lgssc unregistered [ 9536.862815] LNet: 248749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9536.867322] LNetError: 248749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9536.881362] LNet: Removed LNI 192.168.202.137@tcp [ 9537.507526] Key type .llcrypt unregistered [ 9537.509247] Key type ._llcrypt unregistered [ 9555.790700] Key type ._llcrypt registered [ 9555.793322] Key type .llcrypt registered [ 9555.862807] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 9566.394952] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9567.123118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9567.157504] alg: No test for adler32 (adler32-zlib) [ 9568.137555] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 9568.327058] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 9569.976157] Key type lgssc registered [ 9570.785655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9601.867222] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9609.526604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9609.543834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9610.755359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9610.779191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9610.842360] Lustre: lustre-MDT0000: new disk, initializing [ 9610.905270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9610.929802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9614.133414] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9621.999973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9622.064508] Lustre: 252794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9622.100380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9622.103474] Lustre: Skipped 1 previous similar message [ 9622.169179] Lustre: lustre-MDT0001: new disk, initializing [ 9622.239496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9622.281228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9622.292526] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9625.765903] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9629.770738] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9634.856968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9635.109399] Lustre: lustre-OST0000: new disk, initializing [ 9635.112716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9635.118611] Lustre: 254423:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9635.207934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9639.341375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9639.349189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9639.485848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9640.354981] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9648.625711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9648.749792] Lustre: lustre-OST0001: new disk, initializing [ 9648.754231] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9648.766139] Lustre: 255294:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9648.907871] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9653.598516] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9654.279315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9654.286905] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9654.326192] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9662.926049] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9667.242772] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9677.445919] Lustre: server umount lustre-MDT0000 complete [ 9678.821804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9678.829515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9679.843531] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9679.856147] Lustre: Skipped 1 previous similar message [ 9680.232513] LustreError: 255293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306617 with bad export cookie 15855657334650251228 [ 9680.236461] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9680.249774] LustreError: 255293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9680.433612] Lustre: server umount lustre-MDT0001 complete [ 9692.214056] Lustre: server umount lustre-OST0000 complete [ 9705.291814] Lustre: server umount lustre-OST0001 complete [ 9714.505240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9720.000366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9724.887658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9731.334911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9743.456551] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9749.550635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9749.563216] Lustre: lustre-MDT0000: reset Object Index mappings [ 9749.831917] LustreError: 260300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9749.849960] LustreError: 260300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9749.900319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9752.457243] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9755.109962] LustreError: 260301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9757.567549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9757.582586] Lustre: lustre-MDT0001: reset Object Index mappings [ 9760.476631] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9762.240015] Lustre: 261442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9766.249034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9766.452772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9766.456434] Lustre: Skipped 1 previous similar message [ 9767.463377] LustreError: 261795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9769.946850] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9772.517655] LustreError: 261797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9775.365932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9776.574473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9776.580550] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9779.946779] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9785.173593] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9787.564500] Lustre: 263312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9791.972113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9791.977539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9791.978239] Lustre: Skipped 3 previous similar messages [ 9791.986404] Lustre: Skipped 3 previous similar messages [ 9796.163970] Lustre: server umount lustre-MDT0000 complete [ 9797.093216] LustreError: 260300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9797.104665] LustreError: 260300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9798.332987] LustreError: 260280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306735 with bad export cookie 15855657334650258018 [ 9798.334536] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9798.340884] LustreError: 260280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9798.552090] Lustre: server umount lustre-MDT0001 complete [ 9811.520280] Lustre: server umount lustre-OST0000 complete [ 9823.161662] Lustre: server umount lustre-OST0001 complete [ 9835.357330] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 09:39:30 (1789306770) [ 9846.338338] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 9852.516671] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9879.086947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9879.278220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9879.300763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9879.372054] Lustre: lustre-MDT0000: new disk, initializing [ 9879.428544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9879.434067] Lustre: Skipped 1 previous similar message [ 9879.443895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9882.969186] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9890.461754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9890.528583] Lustre: 267561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9890.569687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9890.577229] Lustre: Skipped 1 previous similar message [ 9890.664157] Lustre: lustre-MDT0001: new disk, initializing [ 9890.803101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9890.815660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9893.416310] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9899.464597] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9903.160275] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9907.733629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9907.909594] Lustre: lustre-OST0000: new disk, initializing [ 9907.912339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9907.919740] Lustre: 269529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9909.568672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9909.579510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9909.692623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9911.672073] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9917.430090] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9920.186870] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9920.328754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9922.650372] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9922.797489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9925.091190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9925.092166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9925.098137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9925.111532] Lustre: Skipped 2 previous similar messages [ 9930.210343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9930.212988] Lustre: Skipped 2 previous similar messages [ 9935.332984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9935.337033] Lustre: Skipped 1 previous similar message [ 9938.400157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9938.478909] Lustre: server umount lustre-OST0000 complete [ 9941.987222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9941.990907] LustreError: Skipped 1 previous similar message [ 9941.995713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9942.003474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9946.683698] Lustre: server umount lustre-MDT0000 complete [ 9948.435371] LustreError: 268358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306885 with bad export cookie 15855657334650260104 [ 9948.439212] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9948.442622] LustreError: 268358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9948.592693] Lustre: server umount lustre-MDT0001 complete [ 9957.736649] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9963.779932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9963.986898] LustreError: 272474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9964.009683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9964.012703] Lustre: Skipped 2 previous similar messages [ 9966.191572] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9970.684677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9973.051545] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9974.550625] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9978.035550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9981.659753] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9986.294354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9986.396323] Lustre: lustre-OST0001: new disk, initializing [ 9986.398442] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9986.402169] Lustre: 274780:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9987.938115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9987.942338] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9987.995681] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9989.730655] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9996.069387] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9998.071201] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10001.010824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10001.083145] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10004.452196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10004.461770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10004.470494] Lustre: Skipped 4 previous similar messages [10004.476711] Lustre: Skipped 5 previous similar messages [10010.232072] Lustre: server umount lustre-MDT0000 complete [10011.897364] LustreError: 272455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306949 with bad export cookie 15855657334650260979 [10011.899227] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10011.903740] LustreError: 272455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10012.043542] Lustre: server umount lustre-MDT0001 complete [10023.912433] Lustre: server umount lustre-OST0000 complete [10036.179647] Lustre: server umount lustre-OST0001 complete [10045.511308] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10050.825614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10051.067775] LustreError: 277833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10051.077686] LustreError: 277833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10051.117273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10051.120335] Lustre: Skipped 3 previous similar messages [10053.412140] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10058.127130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10060.734866] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10062.383383] Lustre: 278973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10065.994695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10069.402546] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10074.541184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10076.794289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10076.839707] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10078.110971] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10082.911241] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10084.834182] Lustre: 280851:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10087.577343] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10087.644408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10093.001721] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 09:43:48 (1789307028) [10107.875352] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.876572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10107.885062] Lustre: Skipped 2 previous similar messages [10107.887713] Lustre: Skipped 5 previous similar messages [10111.677959] Lustre: server umount lustre-MDT0000 complete [10113.000142] LustreError: 277829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10113.013930] LustreError: 277829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10113.686366] LustreError: 277814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307050 with bad export cookie 15855657334650267300 [10113.690940] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10113.692948] LustreError: 277814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10119.922410] Lustre: server umount lustre-MDT0001 complete [10128.005843] Lustre: server umount lustre-OST0000 complete [10144.225823] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10144.321205] Lustre: server umount lustre-OST0001 complete [10147.297400] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [10151.229963] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10174.802722] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10180.834325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10180.955172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10180.975855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10181.026462] Lustre: lustre-MDT0000: new disk, initializing [10181.069963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10181.073312] Lustre: Skipped 3 previous similar messages [10181.080417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10183.033203] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10188.727225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10188.803927] Lustre: 285087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10189.005504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10191.110385] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10197.552872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10197.699732] Lustre: 286716:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10199.852353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10200.889630] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10206.879260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10206.998914] Lustre: 287586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10208.785285] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10210.453220] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10216.212244] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10217.938125] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10217.944722] Lustre: Skipped 1 previous similar message [10224.092363] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 09:45:59 (1789307159) [10225.123717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10225.127141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10225.137656] Lustre: Skipped 3 previous similar messages [10225.142286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10225.147458] Lustre: Skipped 4 previous similar messages [10231.322348] Lustre: server umount lustre-MDT0000 complete [10233.285913] LustreError: 285077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307170 with bad export cookie 15855657334650275182 [10233.296024] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10233.297206] LustreError: 285077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10233.454640] Lustre: server umount lustre-MDT0001 complete [10245.709973] Lustre: server umount lustre-OST0000 complete [10257.587468] Lustre: server umount lustre-OST0001 complete [10262.085266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10262.365347] LustreError: 289829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10262.380223] LustreError: 289829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10264.469956] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10269.009775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10271.006647] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10274.391678] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10277.154835] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10280.726718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10284.169197] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10288.230587] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10305.782086] loop1: detected capacity change from 0 to 204800000 [10305.916193] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10305.924123] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10305.929426] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10308.743297] loop1: detected capacity change from 0 to 204800000 [10311.865467] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10311.912320] LustreError: 289818:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs14854: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10311.918498] LustreError: 289818:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs14854 [10311.921089] LustreError: 289818:0:(mgs_handler.c:517:mgs_target_reg()) fs14854-MDT0001: Error checking target index: rc = -13 [10311.924891] LustreError: fs14854-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10311.929447] LustreError: 293563:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10311.932109] LustreError: 293563:0:(tgt_mount.c:2132:server_put_super()) no obd fs14854-MDT0001 [10311.935209] LustreError: 293563:0:(tgt_mount.c:116:server_deregister_mount()) fs14854-MDT0001 not registered [10311.969982] Lustre: server umount fs14854-MDT0001 complete [10311.972215] LustreError: 293563:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10316.644084] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10316.691297] Lustre: Setting parameter fs14854-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs14854-MDT0001 [10316.720884] Lustre: srv-fs14854-MDT0001: No data found on store. Initialize space. [10316.723269] Lustre: Skipped 4 previous similar messages [10316.770367] Lustre: fs14854-MDT0001: new disk, initializing [10316.772877] Lustre: Skipped 3 previous similar messages [10316.848287] Lustre: srv-fs14854-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10316.851054] Lustre: 294029:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14854-MDT0001-mdtlov: can't create new llog: rc = -115 [10316.854732] LustreError: 294029:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14854-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10319.264822] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10323.358076] Lustre: server umount fs14854-MDT0001 complete [10330.193883] loop1: detected capacity change from 0 to 204800000 [10332.823742] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10332.990768] Lustre: 295595:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14854-MDT0001-mdtlov: can't create new llog: rc = -115 [10332.997219] LustreError: 295595:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14854-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10334.957434] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10337.267534] Lustre: server umount fs14854-MDT0001 complete [10344.766477] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.37@tcp (stopping) [10344.770478] Lustre: Skipped 4 previous similar messages [10345.446453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10345.456331] Lustre: Skipped 5 previous similar messages [10348.528852] Lustre: server umount lustre-MDT0000 complete [10353.181468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10353.248492] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10355.484289] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10358.766417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10359.453692] loop1: detected capacity change from 0 to 204800000 [10362.246516] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10362.418777] Lustre: 298038:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14854-MDT0001-mdtlov: can't create new llog: rc = -115 [10362.423921] LustreError: 298038:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14854-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10364.315472] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10366.548516] Lustre: server umount fs14854-MDT0001 complete [10376.160904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10381.209960] Lustre: server umount lustre-OST0000 complete [10384.354751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10384.359738] LustreError: Skipped 2 previous similar messages [10397.152226] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10397.269233] Lustre: server umount lustre-MDT0000 complete [10399.062475] LustreError: 289858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307336 with bad export cookie 15855657334650277856 [10399.067755] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10399.072446] LustreError: 289858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10399.252583] Lustre: server umount lustre-MDT0001 complete [10404.139520] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [10405.826213] Key type lgssc unregistered [10406.019054] LNet: 300380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10406.023831] LNetError: 300380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10406.034866] LNet: Removed LNI 192.168.202.137@tcp [10406.455135] Key type .llcrypt unregistered [10406.457041] Key type ._llcrypt unregistered [10416.359256] Key type ._llcrypt registered [10416.362627] Key type .llcrypt registered [10416.460971] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10417.069480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10417.079480] alg: No test for adler32 (adler32-zlib) [10418.006067] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [10418.141267] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [10419.752166] Key type lgssc registered [10420.403062] Lustre: Echo OBD driver; http://www.lustre.org/ [10424.482578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10424.495740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10425.758248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10427.942719] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10432.360123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10432.549392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10434.826887] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10438.516510] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10441.121792] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10444.445352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10444.670947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10447.589342] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10450.789354] LustreError: 303454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10450.799257] LustreError: 303454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10451.450597] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10452.692119] LustreError: 303455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10456.035169] LustreError: 304074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10462.184202] LustreError: 304126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10462.200796] LustreError: 304126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10464.114146] loop1: detected capacity change from 0 to 204800000 [10464.187636] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10464.194592] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10464.202786] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10466.572474] loop1: detected capacity change from 0 to 204800000 [10467.303135] LustreError: 304126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10467.314097] LustreError: 304126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10468.995399] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10469.069604] Lustre: fs5495-OST0000: new disk, initializing [10469.071964] Lustre: srv-fs5495-OST0000: No data found on store. Initialize space. [10469.075757] Lustre: 305358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10469.109339] Lustre: fs5495-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10471.560481] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10475.812538] Lustre: server umount fs5495-OST0000 complete [10477.538334] LustreError: 304126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10477.544718] LustreError: 304126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10480.970672] loop1: detected capacity change from 0 to 204800000 [10483.353156] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10483.430948] Lustre: fs5495-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10486.016884] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10489.479318] loop2: detected capacity change from 0 to 204800000 [10489.569732] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10489.574919] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10489.581177] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10491.781782] loop2: detected capacity change from 0 to 204800000 [10493.746170] LustreError: 304074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10493.754746] LustreError: 304074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10494.551619] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10494.602798] LustreError: fs5495-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10494.611192] LustreError: fs5495-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10494.617279] LustreError: 308393:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10494.621686] LustreError: 308393:0:(tgt_mount.c:2132:server_put_super()) no obd fs5495-OST0001 [10494.623942] LustreError: 308393:0:(tgt_mount.c:116:server_deregister_mount()) fs5495-OST0001 not registered [10494.638363] Lustre: server umount fs5495-OST0001 complete [10494.640441] LustreError: 308393:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10500.056657] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10500.142949] Lustre: fs5495-OST0001: new disk, initializing [10500.147213] Lustre: srv-fs5495-OST0001: No data found on store. Initialize space. [10500.152975] Lustre: 308944:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10500.190215] Lustre: fs5495-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10503.267252] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10508.008887] Lustre: server umount fs5495-OST0000 complete [10512.431288] Lustre: server umount fs5495-OST0001 complete [10517.654682] loop1: detected capacity change from 0 to 204800000 [10520.040513] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10520.137016] Lustre: fs5495-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10522.639958] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10525.605133] loop2: detected capacity change from 0 to 204800000 [10528.141625] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10528.741557] LustreError: 304126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10528.753773] LustreError: 304126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [10530.913068] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10533.874560] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10534.789306] Lustre: server umount fs5495-OST0000 complete [10539.116724] Lustre: server umount fs5495-OST0001 complete [10544.105262] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10548.193962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10548.197571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10548.203068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10549.218582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10549.220555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10550.614909] Lustre: server umount lustre-OST0000 complete [10554.337666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10554.339289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10554.343723] Lustre: Skipped 1 previous similar message [10554.349537] Lustre: Skipped 2 previous similar messages [10558.314173] Lustre: server umount lustre-MDT0000 complete [10559.691439] LustreError: 301608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307496 with bad export cookie 17731204837827784878 [10559.693884] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10559.696386] LustreError: 301608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10559.817191] Lustre: server umount lustre-MDT0001 complete [10563.893616] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [10565.044891] Key type lgssc unregistered [10565.176502] LNet: 314854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10565.179960] LNetError: 314854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10565.191582] LNet: Removed LNI 192.168.202.137@tcp [10565.530132] Key type .llcrypt unregistered [10565.531546] Key type ._llcrypt unregistered [10579.133585] Key type ._llcrypt registered [10579.134822] Key type .llcrypt registered [10579.184612] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [10585.912803] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10586.315080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10586.325466] alg: No test for adler32 (adler32-zlib) [10587.181181] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [10587.273221] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [10588.856269] Key type lgssc registered [10589.258657] Lustre: Echo OBD driver; http://www.lustre.org/ [10602.250125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10602.267526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10603.365883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10603.377667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10603.421933] Lustre: lustre-MDT0000: new disk, initializing [10603.446747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10603.452740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10604.759993] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10608.900184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10608.944055] Lustre: 318681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10608.971727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10608.974404] Lustre: Skipped 1 previous similar message [10609.019169] Lustre: lustre-MDT0001: new disk, initializing [10609.055122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10609.068929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10609.073700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10610.524081] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10614.148761] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10615.889406] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10618.207262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10618.313229] Lustre: lustre-OST0000: new disk, initializing [10618.315584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10618.317988] Lustre: 320646:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10618.347191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10619.065502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10619.069862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10619.103794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10620.314398] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10623.930746] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10624.603543] Lustre: Mounted lustre-client - version 2.17.57_108_gfed9771 [10624.928587] Lustre: Unmounted lustre-client [10629.602678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10629.603987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10629.605345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10629.608385] Lustre: Skipped 1 previous similar message [10633.814943] Lustre: server umount lustre-OST0000 complete [10639.842723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10639.842869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10639.843686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10639.843694] Lustre: Skipped 1 previous similar message [10639.847361] Lustre: Skipped 1 previous similar message [10639.853807] LustreError: Skipped 1 previous similar message [10641.239925] Lustre: server umount lustre-MDT0000 complete [10642.504261] LustreError: 318673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307579 with bad export cookie 14618829834072034568 [10642.508199] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10642.632614] Lustre: server umount lustre-MDT0001 complete [10646.260960] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [10647.323883] Key type lgssc unregistered [10680.224377] LNet: 322804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10680.227248] LNetError: 322804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10753.958327] LNet: Removed LNI 192.168.202.137@tcp [10754.250247] Key type .llcrypt unregistered [10754.251338] Key type ._llcrypt unregistered [10760.422093] Key type ._llcrypt registered [10760.423245] Key type .llcrypt registered [10760.461774] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [10766.803241] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10767.240755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10767.291553] alg: No test for adler32 (adler32-zlib) [10768.161566] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [10768.250486] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [10769.832210] Key type lgssc registered [10770.208223] Lustre: Echo OBD driver; http://www.lustre.org/ [10779.918327] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 09:55:15 (1789307715) [10782.180879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10782.187489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10783.293213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10783.308943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10783.347333] Lustre: lustre-MDT0000: new disk, initializing [10783.367871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10783.374408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10784.765783] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10788.923414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10788.968363] Lustre: 326000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10788.992630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10788.995386] Lustre: Skipped 1 previous similar message [10789.039350] Lustre: lustre-MDT0001: new disk, initializing [10789.076500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10789.088949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10789.092165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10790.526311] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10794.473549] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10796.350615] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10798.732524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10798.854680] Lustre: lustre-OST0000: new disk, initializing [10798.856949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10798.859715] Lustre: 327965:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10798.892255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10801.085891] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10804.733614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10804.737563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10804.771188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10804.931869] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10814.946118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10814.947545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10814.949747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10814.954584] Lustre: Skipped 1 previous similar message [10816.806794] Lustre: server umount lustre-OST0000 complete [10820.066027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10820.066121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10820.066898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10820.066903] Lustre: Skipped 1 previous similar message [10820.071339] Lustre: Skipped 1 previous similar message [10824.427910] Lustre: server umount lustre-MDT0000 complete [10825.776318] LustreError: 325991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307762 with bad export cookie 10779962232376693335 [10825.779170] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10825.781044] LustreError: 325991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10825.897395] Lustre: server umount lustre-MDT0001 complete [10829.384301] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [10830.483106] Key type lgssc unregistered [10830.615347] LNet: 329869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10830.617578] LNetError: 329869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10830.626316] LNet: Removed LNI 192.168.202.137@tcp [10830.962422] Key type .llcrypt unregistered [10830.963940] Key type ._llcrypt unregistered [10839.725266] Key type ._llcrypt registered [10839.726353] Key type .llcrypt registered [10839.763693] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10840.077639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10840.084969] alg: No test for adler32 (adler32-zlib) [10840.945793] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [10841.042030] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [10842.624160] Key type lgssc registered [10843.002577] Lustre: Echo OBD driver; http://www.lustre.org/ [10845.665468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10845.672542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10846.823149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10848.212943] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10851.135680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.267240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10852.661840] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10855.198134] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10857.084050] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10859.368300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10859.485317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10861.353503] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10863.695266] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10877.498272] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10878.027032] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10882.017222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10882.019924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10882.024799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10883.041823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10883.044845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10884.407030] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.37@tcp (stopping) [10884.410079] Lustre: Skipped 2 previous similar messages [10885.596910] Lustre: server umount lustre-MDT0000 complete [10886.944797] LustreError: 331096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307824 with bad export cookie 2987015856993611118 [10886.945567] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10886.948510] LustreError: 331096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10888.161521] LustreError: 331138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10888.162773] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10888.168596] LustreError: 331138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10888.173513] Lustre: Skipped 1 previous similar message [10888.179442] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10889.516703] LustreError: 331109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10891.168805] LustreError: 331138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10893.103620] Lustre: server umount lustre-MDT0001 complete [10900.566477] Lustre: server umount lustre-OST0000 complete [10903.768105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10905.287408] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10906.372948] Lustre: 335313:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.137@tcp [10907.177782] Lustre: server umount MGS complete [10910.468177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10910.641409] LustreError: 335882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10910.672163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10912.293187] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10915.512050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10915.665731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10917.315380] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10920.570861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10922.842864] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10925.533608] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10926.035354] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10928.035147] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10928.601919] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10932.192942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10932.195676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10932.200618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10932.203115] Lustre: Skipped 2 previous similar messages [10935.268188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10937.695419] Lustre: server umount lustre-OST0000 complete [10940.385637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10940.390630] Lustre: Skipped 1 previous similar message [10940.393470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10940.396107] Lustre: Skipped 3 previous similar messages [10945.374154] Lustre: server umount lustre-MDT0000 complete [10945.505271] LustreError: 335882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10945.510447] LustreError: 335882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10946.750336] LustreError: 337339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307883 with bad export cookie 2987015856993611944 [10946.751302] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10946.754972] LustreError: 337339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10946.882952] Lustre: server umount lustre-MDT0001 complete [10950.788255] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [10951.906537] Key type lgssc unregistered [10952.038468] LNet: 339313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10952.040939] LNetError: 339313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10952.058360] LNet: Removed LNI 192.168.202.137@tcp [10952.402598] Key type .llcrypt unregistered [10952.404242] Key type ._llcrypt unregistered [10959.454652] Key type ._llcrypt registered [10959.455883] Key type .llcrypt registered [10959.499194] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [10966.039926] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10966.494722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10966.516273] alg: No test for adler32 (adler32-zlib) [10967.392610] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [10967.494212] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [10969.080191] Key type lgssc registered [10969.524848] Lustre: Echo OBD driver; http://www.lustre.org/ [10982.094863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10982.099787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10983.187343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10983.200153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10983.240162] Lustre: lustre-MDT0000: new disk, initializing [10983.266819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10983.273420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10984.736765] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10989.072656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10989.118312] Lustre: 342444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10989.142583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10989.144913] Lustre: Skipped 1 previous similar message [10989.179657] Lustre: lustre-MDT0001: new disk, initializing [10989.212387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10989.228361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10989.232834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10990.630508] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10994.556294] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10996.547142] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10998.961618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10999.069722] Lustre: lustre-OST0000: new disk, initializing [10999.071986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10999.075322] Lustre: 344417:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10999.106560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11001.013445] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11004.782990] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11006.220883] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11007.479377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11007.483260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11007.513952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11008.330537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [11009.750341] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11009.823574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11012.578349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11012.578744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11012.582659] LustreError: Skipped 1 previous similar message [11012.589034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11012.593348] Lustre: Skipped 1 previous similar message [11016.595450] Lustre: server umount lustre-OST0000 complete [11020.256800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11020.259798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11020.265092] Lustre: Skipped 1 previous similar message [11020.267069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11022.817109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11022.819379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11024.031952] Lustre: server umount lustre-MDT0000 complete [11025.423237] LustreError: 342439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307962 with bad export cookie 2641013147433467570 [11025.426191] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11025.427676] LustreError: 342439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11025.544922] Lustre: server umount lustre-MDT0001 complete [11028.781517] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 09:59:24 (1789307964) [11034.068755] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [11037.528753] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11049.496900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11049.573827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11049.584422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11049.618139] Lustre: lustre-MDT0000: new disk, initializing [11049.638630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11049.645575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11051.012598] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11055.093417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11055.136101] Lustre: 349177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11055.155546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11055.157229] Lustre: Skipped 1 previous similar message [11055.190867] Lustre: lustre-MDT0001: new disk, initializing [11055.220424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11055.231080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11055.235359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11056.478048] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11060.061253] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11061.844963] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11064.024327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11064.116137] Lustre: lustre-OST0000: new disk, initializing [11064.117554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11064.119653] Lustre: 351146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11064.149216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11066.012844] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11068.919170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11068.922440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11068.952989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11069.638063] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11084.257844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11084.260426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11084.261942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11084.264804] Lustre: Skipped 1 previous similar message [11084.266548] Lustre: Skipped 1 previous similar message [11086.101676] Lustre: server umount lustre-OST0000 complete [11089.377684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11089.378435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11089.382110] Lustre: Skipped 1 previous similar message [11089.383892] Lustre: Skipped 1 previous similar message [11093.596417] Lustre: server umount lustre-MDT0000 complete [11094.863411] LustreError: 349973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308032 with bad export cookie 2641013147433468438 [11094.865887] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11094.868247] LustreError: 349973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11094.992222] Lustre: server umount lustre-MDT0001 complete [11098.823846] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11099.928271] Key type lgssc unregistered [11100.066410] LNet: 353293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11100.068930] LNetError: 353293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11100.080326] LNet: Removed LNI 192.168.202.137@tcp [11100.409221] Key type .llcrypt unregistered [11100.410332] Key type ._llcrypt unregistered [11132.594438] Key type ._llcrypt registered [11132.595919] Key type .llcrypt registered [11132.634211] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11132.731532] Key type .llcrypt unregistered [11132.732996] Key type ._llcrypt unregistered [11159.286926] Key type ._llcrypt registered [11159.288092] Key type .llcrypt registered [11159.326276] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11159.433210] Key type .llcrypt unregistered [11159.434257] Key type ._llcrypt unregistered [11170.177113] Key type ._llcrypt registered [11170.178128] Key type .llcrypt registered [11170.219413] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11170.329264] Key type .llcrypt unregistered [11170.330307] Key type ._llcrypt unregistered [11180.823491] Key type ._llcrypt registered [11180.824307] Key type .llcrypt registered [11180.853465] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11180.936317] Key type .llcrypt unregistered [11180.937250] Key type ._llcrypt unregistered [11206.883944] Key type ._llcrypt registered [11206.884975] Key type .llcrypt registered [11206.924126] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11207.023253] Key type .llcrypt unregistered [11207.024231] Key type ._llcrypt unregistered [11217.467791] Key type ._llcrypt registered [11217.469338] Key type .llcrypt registered [11217.508245] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11217.608447] Key type .llcrypt unregistered [11217.610045] Key type ._llcrypt unregistered [11228.865815] Key type ._llcrypt registered [11228.866696] Key type .llcrypt registered [11228.896766] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11228.987293] Key type .llcrypt unregistered [11228.988213] Key type ._llcrypt unregistered [11239.366536] Key type ._llcrypt registered [11239.367711] Key type .llcrypt registered [11239.407207] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11239.525363] Key type .llcrypt unregistered [11239.526517] Key type ._llcrypt unregistered [11250.961558] Key type ._llcrypt registered [11250.962558] Key type .llcrypt registered [11250.999581] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11251.114234] Key type .llcrypt unregistered [11251.115606] Key type ._llcrypt unregistered [11265.848836] Key type ._llcrypt registered [11265.849859] Key type .llcrypt registered [11265.886863] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11266.004667] Key type .llcrypt unregistered [11266.006040] Key type ._llcrypt unregistered [11289.021492] Key type ._llcrypt registered [11289.022606] Key type .llcrypt registered [11289.061703] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11289.164357] Key type .llcrypt unregistered [11289.165363] Key type ._llcrypt unregistered [11300.386710] Key type ._llcrypt registered [11300.387724] Key type .llcrypt registered [11300.421319] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11300.515403] Key type .llcrypt unregistered [11300.516958] Key type ._llcrypt unregistered [11311.182664] Key type ._llcrypt registered [11311.183678] Key type .llcrypt registered [11311.215943] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11311.309712] Key type .llcrypt unregistered [11311.310688] Key type ._llcrypt unregistered [11328.965596] Key type ._llcrypt registered [11328.966781] Key type .llcrypt registered [11329.013307] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [11335.209904] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11335.581034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11335.652091] alg: No test for adler32 (adler32-zlib) [11336.527190] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [11336.625207] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [11338.200190] Key type lgssc registered [11338.564918] Lustre: Echo OBD driver; http://www.lustre.org/ [11350.132597] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11350.139619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11351.221097] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [11351.232462] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [11351.270058] Lustre: tst250fs-MDT0000: new disk, initializing [11351.296367] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11351.303387] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11352.566529] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11356.698125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11356.740217] Lustre: 373150:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11356.761410] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [11356.763465] Lustre: Skipped 1 previous similar message [11356.799523] Lustre: tst250fs-MDT0001: new disk, initializing [11356.830535] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11356.841831] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11356.844859] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11358.094200] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11360.313310] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11362.449905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11362.544577] Lustre: tst250fs-OST0000: new disk, initializing [11362.547366] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [11362.551066] Lustre: 374778:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11362.579799] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11364.500256] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11368.446524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11368.502517] Lustre: tst250fs-OST0001: new disk, initializing [11368.504584] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [11368.506815] Lustre: 375638:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11368.538671] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11368.657751] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11368.660903] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11368.665394] Lustre: Skipped 1 previous similar message [11368.714695] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11368.716741] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11370.455820] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11373.013355] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [11384.291393] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11384.292724] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11384.295653] Lustre: Skipped 3 previous similar messages [11384.297411] Lustre: Skipped 2 previous similar messages [11389.214139] Lustre: server umount tst250fs-MDT0000 complete [11390.424216] LustreError: 373143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308327 with bad export cookie 5462226877573790951 [11390.427345] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11390.427813] LustreError: 373143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11390.538686] Lustre: server umount tst250fs-MDT0001 complete [11401.728911] Lustre: server umount tst250fs-OST0000 complete [11413.206354] Lustre: server umount tst250fs-OST0001 complete [11418.764657] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [11421.164449] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11431.024839] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 10:06:07 (1789308367) [11431.481766] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [11436.728578] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [11439.948945] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11449.909306] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11143 sec ========= 10:06:25 (1789308385) [11450.432295] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:06:26 (1789308386) === [11451.449941] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:06:27 (1789308387) === [11463.973912] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11464.892383] Key type lgssc unregistered [11465.025278] LNet: 384756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11465.027195] LNetError: 384756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11465.035230] LNet: Removed LNI 192.168.202.137@tcp [11465.316121] Key type .llcrypt unregistered [11465.317068] Key type ._llcrypt unregistered